Sep 18 13:56:36.232582 ip-10-0-139-55 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 13:56:36.232593 ip-10-0-139-55 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 13:56:36.232600 ip-10-0-139-55 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 13:56:36.232828 ip-10-0-139-55 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 13:56:45.082639 ip-10-0-139-55 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 18 13:56:45.082655 ip-10-0-139-55 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ff47365f36bb4717b39892fabc92f385 --
Sep 18 13:58:30.682295 ip-10-0-139-55 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 13:58:31.046436 ip-10-0-139-55 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:58:31.046436 ip-10-0-139-55 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 13:58:31.046436 ip-10-0-139-55 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:58:31.046436 ip-10-0-139-55 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 13:58:31.046436 ip-10-0-139-55 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:58:31.048745 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.048586    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 13:58:31.052252 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052238    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:31.052252 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052252    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052256    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052264    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052267    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052270    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052273    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052276    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052279    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052281    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052284    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052287    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052289    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052292    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052295    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052297    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052300    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052302    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052305    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052307    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052310    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:31.052314 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052312    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052315    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052318    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052320    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052323    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052326    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052329    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052331    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052333    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052336    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052339    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052341    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052344    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052346    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052349    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052352    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052354    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052357    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052359    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052361    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:31.052908 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052364    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052368    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052371    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052374    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052376    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052379    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052382    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052384    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052387    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052389    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052392    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052394    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052396    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052399    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052403    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052406    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052409    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052411    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052414    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052416    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:31.053519 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052419    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052422    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052424    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052427    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052429    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052432    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052434    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.052468    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053433    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053453    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053460    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053465    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053470    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053475    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053480    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053485    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053490    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053500    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053505    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:31.054056 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053516    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053520    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053525    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053529    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053533    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053538    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053542    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053546    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053550    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053554    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053558    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053562    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053570    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053574    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053578    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053582    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053586    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053590    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:31.054506 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.053594    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:31.055657 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055643    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:31.055657 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055656    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055659    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055662    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055665    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055668    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055670    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055674    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055676    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055679    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055681    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055684    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055686    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055689    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055691    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055693    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055696    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055699    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055701    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055706    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055710    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:31.055713 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055712    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055715    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055718    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055721    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055724    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055727    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055730    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055733    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055736    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055738    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055741    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055744    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055746    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055749    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055751    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055753    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055756    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055758    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055760    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055763    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:31.056205 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055765    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055768    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055770    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055773    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055776    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055778    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055781    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055785    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055788    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055790    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055793    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055795    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055797    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055800    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055804    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055806    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055809    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055811    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055814    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:31.056672 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055817    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055819    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055822    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055824    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055827    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055830    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055832    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055835    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055838    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055840    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055842    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055845    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055847    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055850    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055852    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055854    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055857    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055859    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055864    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:31.057124 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055867    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055870    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055872    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055875    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055878    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055880    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055883    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055885    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055889    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055891    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055894    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055897    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055899    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055901    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055904    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055906    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055909    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055911    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055914    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.055916    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:31.057585 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056277    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056282    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056285    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056289    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056292    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056295    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056298    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056300    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056303    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056305    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056308    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056310    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056312    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056315    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056317    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056320    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056322    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056326    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056330    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:31.058054 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056332    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056336    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056340    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056343    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056345    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056348    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056351    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056353    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056356    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056358    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056361    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056364    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056366    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056369    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056371    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056373    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056375    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056378    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056380    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056383    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:31.058511 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056385    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056388    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056391    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056393    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056396    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056398    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056400    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056404    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056406    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056409    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056411    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056414    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056417    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056420    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056423    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056425    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056428    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056431    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056435    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:31.058988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056438    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056440    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056443    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056446    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056448    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056451    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056453    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056456    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056458    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056460    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056463    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056465    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056468    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056471    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056474    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056476    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056478    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056481    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056483    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056486    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:31.059436 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056488    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056490    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056493    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056496    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056498    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056501    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056504    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056507    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056509    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056511    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056514    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056516    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056518    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056521    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056523    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056526    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056528    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056530    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056533    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056535    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:31.059952 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.056538    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056615    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056663    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056669    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056675    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056679    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056682    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056686    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056690    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056693    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056696    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056700    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056703    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056706    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056709    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056712    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056715    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056718    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056721    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056728    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056731    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056734    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056737    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056740    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 13:58:31.060421 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056744    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056747    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056750    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056753    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056755    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056758    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056761    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056764    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056767    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056771    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056775    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056777    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056780    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056783    2563 flags.go:64] FLAG: --enable-server="true"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056786    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056790    2563 flags.go:64] FLAG: --event-burst="100"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056793    2563 flags.go:64] FLAG: --event-qps="50"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056796    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056799    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056802    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056805    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056808    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056811    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056814    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056816    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 13:58:31.060979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056819    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056823    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056825    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056829    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056832    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056835    2563 flags.go:64] FLAG: --feature-gates=""
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056840    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056843    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056846    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056849    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056852    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056855    2563 flags.go:64] FLAG: --help="false"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056859    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056862    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056865    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056867    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056871    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056874    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056877    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056879    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056882    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056885    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056887    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056890    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 13:58:31.061573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056893    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056896    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056899    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056902    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056904    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056907    2563 flags.go:64] FLAG: --lock-file=""
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056910    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056913    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056915    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056922    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056925    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056939    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056943    2563 flags.go:64] FLAG: --logging-format="text"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056946    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056949    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056952    2563 flags.go:64] FLAG: --manifest-url=""
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056955    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056959    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056962    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056966    2563 flags.go:64] FLAG: --max-pods="110"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056970    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056973    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056976    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056979    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056982    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 13:58:31.062151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056984    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056987    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056995    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.056998    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057000    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057003    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057006    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057012    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057014    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057017    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057020    2563 flags.go:64] FLAG: --port="10250"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057023    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057026    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06681c7760305c4cf"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057029    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057032    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057035    2563 flags.go:64] FLAG: --register-node="true"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057038    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057047    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057049    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057052    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057055    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057059    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057062    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057065    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057067    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 13:58:31.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057070    2563 flags.go:64] FLAG: --runonce="false"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057073    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057077    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057080    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057083    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057086    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057089    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057091    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057094    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057097    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057100    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057103    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057106    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057109    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057112    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057114    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057119    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057122    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057125    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057129    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057132    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057135    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057138    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057140    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057145    2563 flags.go:64] FLAG: --v="2"
Sep 18 13:58:31.063359 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057148    2563 flags.go:64] FLAG: --version="false"
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057152    2563 flags.go:64] FLAG: --vmodule=""
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057155    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.057159    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057263    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057267    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057270    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057273    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057275    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057280    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057283    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057285    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057288    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057291    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057293    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057296    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057299    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057301    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057304    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057307    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:31.063988 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057309    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057312    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057314    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057317    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057320    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057322    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057325    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057327    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057330    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057332    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057335    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057338    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057342    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057346    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057349    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057352    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057355    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057358    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057362    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:31.064457 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057365    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057367    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057371    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057376    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057379    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057382    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057385    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057388    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057391    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057394    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057397    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057400    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057402    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057405    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057408    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057410    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057413    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057416    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057418    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:31.064967 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057421    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057423    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057426    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057429    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057431    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057435    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057438    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057441    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057443    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057446    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057448    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057452    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057454    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057457    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057460    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057463    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057466    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057468    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057471    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057473    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:31.065429 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057476    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057478    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057481    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057483    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057486    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057489    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057491    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057494    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057497    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057501    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057503    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057506    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057508    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057511    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057513    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057516    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057518    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057523    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057525    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:31.065903 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057528    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057531    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057533    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057536    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057538    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.057541    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.058028    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.064435    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.064450    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.065179    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.065188    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065239    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065244    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065247    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065251    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:31.066382 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065255    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065258    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065260    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065263    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065265    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065268    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065271    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065274    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065277    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065280    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065282    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065285    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065287    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065289    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065292    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065295    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065298    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065301    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065303    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:31.066774 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065305    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065308    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065311    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065313    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065315    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065320    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065324    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065328    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065330    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065333    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065336    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065338    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065341    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065344    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065347    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065350    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065352    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065355    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065357    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065360    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:31.067236 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065363    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065365    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065368    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065370    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065373    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065375    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065378    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065380    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065382    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065385    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065388    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065390    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065392    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065395    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065397    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065400    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065403    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065405    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065409    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065412    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:31.067708 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065415    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065417    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065420    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065422    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065425    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065427    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065430    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065433    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065436    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065438    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065441    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065443    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065446    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065449    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065451    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065454    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065456    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065459    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065461    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065464    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:31.068188 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065466    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065468    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065471    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065473    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065477    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065480    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065483    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065486    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065489    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065493    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065497    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065500    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065503    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065506    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065509    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:31.068652 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065512    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.065517    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065618    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065622    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065625    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065628    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065631    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065634    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065636    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065639    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065642    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065644    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065647    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065649    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065652    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:58:31.069077 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065654    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065657    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065659    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065662    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065664    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065667    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065669    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065672    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065675    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065677    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065680    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065684    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065687    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065691    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065694    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065697    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065700    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065702    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065705    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:58:31.069434 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065707    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065711    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065714    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065718    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065720    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065723    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065725    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065728    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065730    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065733    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065735    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065737    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065740    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065742    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065745    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065747    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065750    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065752    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065755    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065757    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:58:31.069873 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065760    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065762    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065765    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065767    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065770    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065773    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065775    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065777    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065780    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065783    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065785    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065787    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065790    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065792    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065795    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065797    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065800    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065802    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065805    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065807    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:58:31.070358 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065810    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065812    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065815    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065817    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065820    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065822    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065824    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065827    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065829    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065832    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065834    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065837    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065839    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065842    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065844    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065846    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065849    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065851    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065854    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:58:31.070813 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065857    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065859    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065861    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065865    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065867    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065870    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065873    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:31.065877    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.065882    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.065981    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.067704    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.068610    2563 server.go:1019] "Starting client certificate rotation"
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.068703    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:58:31.071255 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.068762    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:58:31.071557 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.070829    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:58:31.074968 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.074315    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:58:31.085209 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.085192    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 18 13:58:31.086706 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.086693    2563 log.go:25] "Validated CRI v1 image API"
Sep 18 13:58:31.087482 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.087467    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 13:58:31.091125 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.091105    2563 fs.go:135] Filesystem UUIDs: map[1b482509-c2f1-4266-bac0-10cda33b74e3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a3d7cc4e-d7e4-4f2f-9efe-9ed8a3291ef2:/dev/nvme0n1p3]
Sep 18 13:58:31.091206 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.091124    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 13:58:31.096707 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.096612    2563 manager.go:216] Machine: {Timestamp:2026-09-18 13:58:31.095328559 +0000 UTC m=+0.305045996 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200996 MemoryCapacity:33164460032 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ed3d7c42bd3c32f146c9a796de0ba SystemUUID:ec2ed3d7-c42b-d3c3-2f14-6c9a796de0ba BootID:ff47365f-36bb-4717-b398-92fabc92f385 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582230016 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582230016 Type:vfs Inodes:4048396 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ca:1d:4f:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ca:1d:4f:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:74:94:ed:28:26 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164460032 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 13:58:31.096707 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.096706    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 13:58:31.096803 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.096796    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 13:58:31.099565 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.099543    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 13:58:31.099684 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.099568    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-55.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 13:58:31.100659 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.100648    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 13:58:31.100695 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.100662    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 13:58:31.100695 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.100675    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:58:31.101225 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.101215    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:58:31.102070 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.102055    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:58:31.102373 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.102362    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 13:58:31.105746 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.105735    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:58:31.105855 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.105845    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 13:58:31.105893 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.105873    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 13:58:31.105893 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.105882    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 13:58:31.105893 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.105892    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 13:58:31.106022 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.105900    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 18 13:58:31.106022 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.105908    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 13:58:31.106285 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.106271    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:58:31.106330 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.106298    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:58:31.109398 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.109380    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 13:58:31.111660 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.111646    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 13:58:31.111724 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.111675    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 13:58:31.112740 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112728    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 13:58:31.112794 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112743    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 13:58:31.112794 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112748    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 13:58:31.112794 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112753    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 13:58:31.112794 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112762    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 13:58:31.112794 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112767    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 13:58:31.112794 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112781    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 13:58:31.112794 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112790    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 13:58:31.112794 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112797    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 13:58:31.113021 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112803    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 13:58:31.113021 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112812    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 13:58:31.113021 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.112817    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 13:58:31.114136 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.114126    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 13:58:31.114136 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.114136    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 13:58:31.115569 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.115552    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 13:58:31.115619 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.115566    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 13:58:31.117626 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.117614    2563 server.go:1268] "Started kubelet"
Sep 18 13:58:31.117711 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.117690    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 13:58:31.117866 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.117818    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 13:58:31.117944 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.117898    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 13:58:31.118252 ip-10-0-139-55 systemd[1]: Started Kubernetes Kubelet.
Sep 18 13:58:31.119011 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.118994    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 13:58:31.123452 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.123413    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 13:58:31.124151 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.124130    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 13:58:31.127979 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.127953    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-55.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 13:58:31.128594 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.128580    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 13:58:31.128651 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.127874    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-55.ec2.internal.18d66ef77a80be8f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-55.ec2.internal,UID:ip-10-0-139-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-55.ec2.internal,},FirstTimestamp:2026-09-18 13:58:31.117586063 +0000 UTC m=+0.327303500,LastTimestamp:2026-09-18 13:58:31.117586063 +0000 UTC m=+0.327303500,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-55.ec2.internal,}"
Sep 18 13:58:31.129159 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.129145    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 13:58:31.129294 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.129278    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 18 13:58:31.129823 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.129806    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 13:58:31.129823 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.129824    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 13:58:31.129963 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.129809    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 13:58:31.130013 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.129981    2563 factory.go:55] Registering systemd factory
Sep 18 13:58:31.130013 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.129994    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 18 13:58:31.130013 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.129997    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 13:58:31.130013 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.130008    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 13:58:31.130418 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.130211    2563 factory.go:153] Registering CRI-O factory
Sep 18 13:58:31.130418 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.130222    2563 factory.go:223] Registration of the crio container factory successfully
Sep 18 13:58:31.130418 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.130266    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 13:58:31.130418 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.130285    2563 factory.go:103] Registering Raw factory
Sep 18 13:58:31.130418 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.130300    2563 manager.go:1196] Started watching for new ooms in manager
Sep 18 13:58:31.131726 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.131511    2563 manager.go:349] Starting recovery of all containers
Sep 18 13:58:31.133113 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.133088    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 13:58:31.133217 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.133194    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 13:58:31.142165 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.142149    2563 manager.go:354] Recovery completed
Sep 18 13:58:31.145701 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.145689    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:31.147285 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.147260    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:31.147339 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.147294    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:31.147339 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.147304    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:31.147783 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.147767    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 13:58:31.147783 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.147779    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 13:58:31.147783 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.147781    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:31.147908 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.147792    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:58:31.148384 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.148371    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:31.148441 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.148395    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:31.148441 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.148407    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:31.149712 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.149627    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-55.ec2.internal.18d66ef77c45e1ab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-55.ec2.internal,UID:ip-10-0-139-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-55.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-55.ec2.internal,},FirstTimestamp:2026-09-18 13:58:31.147282859 +0000 UTC m=+0.357000295,LastTimestamp:2026-09-18 13:58:31.147282859 +0000 UTC m=+0.357000295,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-55.ec2.internal,}"
Sep 18 13:58:31.149816 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.149804    2563 policy_none.go:49] "None policy: Start"
Sep 18 13:58:31.149848 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.149824    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 13:58:31.149848 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.149839    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 13:58:31.151871 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.151861    2563 policy_none.go:47] "Start"
Sep 18 13:58:31.163857 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.163840    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cchth"
Sep 18 13:58:31.164610 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.164552    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-55.ec2.internal.18d66ef77c4620c7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-55.ec2.internal,UID:ip-10-0-139-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-55.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-55.ec2.internal,},FirstTimestamp:2026-09-18 13:58:31.147299015 +0000 UTC m=+0.357016452,LastTimestamp:2026-09-18 13:58:31.147299015 +0000 UTC m=+0.357016452,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-55.ec2.internal,}"
Sep 18 13:58:31.171994 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.171971    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cchth"
Sep 18 13:58:31.176409 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.176341    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-55.ec2.internal.18d66ef77c464642  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-55.ec2.internal,UID:ip-10-0-139-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-55.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-55.ec2.internal,},FirstTimestamp:2026-09-18 13:58:31.14730861 +0000 UTC m=+0.357026047,LastTimestamp:2026-09-18 13:58:31.14730861 +0000 UTC m=+0.357026047,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-55.ec2.internal,}"
Sep 18 13:58:31.187517 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.187502    2563 manager.go:337] "Starting Device Plugin manager"
Sep 18 13:58:31.211083 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.187548    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 13:58:31.211083 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.187559    2563 server.go:87] "Starting device plugin registration server"
Sep 18 13:58:31.211083 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.187767    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 13:58:31.211083 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.187778    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 13:58:31.211083 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.187881    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 13:58:31.211083 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.187963    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 13:58:31.211083 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.187972    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 13:58:31.211083 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.188377    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 13:58:31.211083 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.188423    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 18 13:58:31.226918 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.226892    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 13:58:31.227972 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.227953    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 13:58:31.228054 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.228009    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 13:58:31.228054 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.228034    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 13:58:31.228132 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.228074    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 13:58:31.230366 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.230351    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:58:31.288299 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.288285    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:31.289033 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.289019    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:31.289100 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.289044    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:31.289100 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.289053    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:31.289100 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.289075    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.300276 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.300228    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.300276 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.300246    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-55.ec2.internal\": node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 18 13:58:31.329107 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.329087    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"]
Sep 18 13:58:31.329190 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.329147    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:31.330667 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.330651    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:31.330747 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.330678    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:31.330747 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.330691    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:31.332864 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.332850    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:31.333001 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.332989    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.333050 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.333014    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:31.333541 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.333525    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:31.333600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.333550    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:31.333600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.333559    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:31.334143 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.334127    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:31.334209 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.334152    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:31.334209 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.334167    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:31.336272 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.336260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.336321 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.336283    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:58:31.336912 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.336895    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:58:31.336984 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.336925    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:58:31.336984 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.336958    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:58:31.348443 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.348427    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-55.ec2.internal\" not found" node="ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.352416 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.352402    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-55.ec2.internal\" not found" node="ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.362406 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.362391    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 18 13:58:31.431134 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.431112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.431230 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.431142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.431230 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.431168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e878fac6c20a59b3277411748de820b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-55.ec2.internal\" (UID: \"3e878fac6c20a59b3277411748de820b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.463200 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.463183    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 18 13:58:31.532153 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.532133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.532224 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.532094    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.532224 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.532200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.532224 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.532219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e878fac6c20a59b3277411748de820b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-55.ec2.internal\" (UID: \"3e878fac6c20a59b3277411748de820b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.532325 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.532251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3e878fac6c20a59b3277411748de820b-config\") pod \"kube-apiserver-proxy-ip-10-0-139-55.ec2.internal\" (UID: \"3e878fac6c20a59b3277411748de820b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.532325 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.532288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f37f127c93297148e7dbac392282653-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal\" (UID: \"7f37f127c93297148e7dbac392282653\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.564260 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.564208    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 18 13:58:31.652020 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.651992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.655927 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.655909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:31.664869 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.664849    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 18 13:58:31.765303 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.765281    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 18 13:58:31.865847 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.865796    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 18 13:58:31.928664 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:31.928643    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:58:31.966235 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:31.966207    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 18 13:58:32.066727 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:32.066703    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-55.ec2.internal\" not found"
Sep 18 13:58:32.068879 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:32.068855    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 13:58:32.069022 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:32.069001    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 13:58:32.069067 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:32.069015    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 13:58:33.194242 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:33.194096    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:58:33.231308 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:33.231282    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 13:53:32 +0000 UTC" deadline="2028-03-29 15:38:45.928736998 +0000 UTC"
Sep 18 13:58:33.231308 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:33.231305    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13393h40m12.697434564s"
Sep 18 13:58:33.282694 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:33.282677    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:33.289012 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:33.288989    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f37f127c93297148e7dbac392282653.slice/crio-f7724dbf552e0c1b93c8a5720ab5e7cf399f092817669e103288014e602e6eeb WatchSource:0}: Error finding container f7724dbf552e0c1b93c8a5720ab5e7cf399f092817669e103288014e602e6eeb: Status 404 returned error can't find the container with id f7724dbf552e0c1b93c8a5720ab5e7cf399f092817669e103288014e602e6eeb
Sep 18 13:58:33.289870 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:33.289842    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e878fac6c20a59b3277411748de820b.slice/crio-cb6f28fc26b54c12160f15ed0af0eedc8b39f6dfa2a465350f2c027737006f72 WatchSource:0}: Error finding container cb6f28fc26b54c12160f15ed0af0eedc8b39f6dfa2a465350f2c027737006f72: Status 404 returned error can't find the container with id cb6f28fc26b54c12160f15ed0af0eedc8b39f6dfa2a465350f2c027737006f72
Sep 18 13:58:33.294456 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:33.294438    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:58:33.296465 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:33.296452    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:58:33.297430 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:33.297415    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal"
Sep 18 13:58:33.304815 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:33.304801    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:58:33.682171 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:33.682151    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:58:34.160334 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.160258    2563 apiserver.go:52] "Watching apiserver"
Sep 18 13:58:34.170031 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.170000    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 13:58:34.172154 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.172107    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-5w4nj","openshift-multus/multus-additional-cni-plugins-ttrvx","openshift-network-operator/iptables-alerter-945p8","kube-system/global-pull-secret-syncer-pp5sf","kube-system/konnectivity-agent-wtgbx","kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal","openshift-dns/node-resolver-plgx7","openshift-multus/network-metrics-daemon-bzklt","openshift-network-diagnostics/network-check-target-j5xvn","openshift-ovn-kubernetes/ovnkube-node-pqnf9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl","openshift-cluster-node-tuning-operator/tuned-d7tbh","openshift-image-registry/node-ca-98pdk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal"]
Sep 18 13:58:34.174621 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.174600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-plgx7"
Sep 18 13:58:34.176717 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.176692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-98pdk"
Sep 18 13:58:34.178175 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.178154    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g7mbb\""
Sep 18 13:58:34.178724 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.178706    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 13:58:34.178807 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.178796    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 13:58:34.179002 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.178986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-945p8"
Sep 18 13:58:34.179938 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.179918    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 13:58:34.180176 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.180159    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 13:58:34.180384 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.180369    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 13:58:34.182120 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.182101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:34.182217 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.182173    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 13:58:34.182217 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.182175    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:34.182608 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.182561    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zlvr9\""
Sep 18 13:58:34.183439 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.182756    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 13:58:34.183439 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.182774    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 13:58:34.183439 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.183240    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h4j87\""
Sep 18 13:58:34.183439 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.183423    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:58:34.183708 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.183687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:34.186008 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.185984    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:34.186098 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.186049    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:34.188091 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.188073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:34.188193 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.188164    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:34.190297 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.190277    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rd465\""
Sep 18 13:58:34.190573 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.190557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.191177 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwkd\" (UniqueName: \"kubernetes.io/projected/8bfae7fb-425a-493c-927c-877fd442c666-kube-api-access-vtwkd\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:34.191258 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94143e9a-35af-4e2f-92fe-e27c520db0ab-tmp-dir\") pod \"node-resolver-plgx7\" (UID: \"94143e9a-35af-4e2f-92fe-e27c520db0ab\") " pod="openshift-dns/node-resolver-plgx7"
Sep 18 13:58:34.191258 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b00ac136-4ce2-4f92-85cb-ffa07460a93c-host-slash\") pod \"iptables-alerter-945p8\" (UID: \"b00ac136-4ce2-4f92-85cb-ffa07460a93c\") " pod="openshift-network-operator/iptables-alerter-945p8"
Sep 18 13:58:34.191258 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9srqp\" (UniqueName: \"kubernetes.io/projected/b00ac136-4ce2-4f92-85cb-ffa07460a93c-kube-api-access-9srqp\") pod \"iptables-alerter-945p8\" (UID: \"b00ac136-4ce2-4f92-85cb-ffa07460a93c\") " pod="openshift-network-operator/iptables-alerter-945p8"
Sep 18 13:58:34.191404 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:34.191404 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svdql\" (UniqueName: \"kubernetes.io/projected/94143e9a-35af-4e2f-92fe-e27c520db0ab-kube-api-access-svdql\") pod \"node-resolver-plgx7\" (UID: \"94143e9a-35af-4e2f-92fe-e27c520db0ab\") " pod="openshift-dns/node-resolver-plgx7"
Sep 18 13:58:34.191404 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6812878a-b037-40b6-89c1-445d49376f65-dbus\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:34.191404 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17faa55e-6e23-473c-828c-8727a86c9977-konnectivity-ca\") pod \"konnectivity-agent-wtgbx\" (UID: \"17faa55e-6e23-473c-828c-8727a86c9977\") " pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:34.191627 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:34.191627 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/336c2a92-68fa-4de6-ba21-fbeab8836c37-host\") pod \"node-ca-98pdk\" (UID: \"336c2a92-68fa-4de6-ba21-fbeab8836c37\") " pod="openshift-image-registry/node-ca-98pdk"
Sep 18 13:58:34.191627 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/336c2a92-68fa-4de6-ba21-fbeab8836c37-serviceca\") pod \"node-ca-98pdk\" (UID: \"336c2a92-68fa-4de6-ba21-fbeab8836c37\") " pod="openshift-image-registry/node-ca-98pdk"
Sep 18 13:58:34.191627 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b00ac136-4ce2-4f92-85cb-ffa07460a93c-iptables-alerter-script\") pod \"iptables-alerter-945p8\" (UID: \"b00ac136-4ce2-4f92-85cb-ffa07460a93c\") " pod="openshift-network-operator/iptables-alerter-945p8"
Sep 18 13:58:34.191627 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6812878a-b037-40b6-89c1-445d49376f65-kubelet-config\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:34.191627 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17faa55e-6e23-473c-828c-8727a86c9977-agent-certs\") pod \"konnectivity-agent-wtgbx\" (UID: \"17faa55e-6e23-473c-828c-8727a86c9977\") " pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:34.191627 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94143e9a-35af-4e2f-92fe-e27c520db0ab-hosts-file\") pod \"node-resolver-plgx7\" (UID: \"94143e9a-35af-4e2f-92fe-e27c520db0ab\") " pod="openshift-dns/node-resolver-plgx7"
Sep 18 13:58:34.191627 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.191604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pppzd\" (UniqueName: \"kubernetes.io/projected/336c2a92-68fa-4de6-ba21-fbeab8836c37-kube-api-access-pppzd\") pod \"node-ca-98pdk\" (UID: \"336c2a92-68fa-4de6-ba21-fbeab8836c37\") " pod="openshift-image-registry/node-ca-98pdk"
Sep 18 13:58:34.193152 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.193116    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 13:58:34.193229 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.193167    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 13:58:34.193327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.193314    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.195730 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.195711    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 13:58:34.195965 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.195948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nb7gc\""
Sep 18 13:58:34.196239 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.196221    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 13:58:34.196491 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.196471    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 13:58:34.196741 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.196721    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zcwsc\""
Sep 18 13:58:34.197384 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.197364    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 13:58:34.198201 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.198184    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.198999 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.198979    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 13:58:34.199546 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.199521    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 13:58:34.200932 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.200910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.202764 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.202744    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-27qgb\""
Sep 18 13:58:34.202764 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.202758    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 13:58:34.203469 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.202941    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 13:58:34.203469 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.202947    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:58:34.203469 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.203333    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.203623 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.203547    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 13:58:34.204541 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.204523    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 13:58:34.204744 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.204727    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 13:58:34.205027 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.205014    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 13:58:34.205346 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.205327    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 13:58:34.205474 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.205456    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 13:58:34.205546 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.205329    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 13:58:34.205600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.205572    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d8w8d\""
Sep 18 13:58:34.209686 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.208843    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:58:34.209686 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.209149    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 13:58:34.209686 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.209418    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v6h9z\""
Sep 18 13:58:34.254550 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.254527    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7rxmk"
Sep 18 13:58:34.264514 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.264459    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7rxmk"
Sep 18 13:58:34.284745 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.284699    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 13:58:34.286112 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.286069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal" event={"ID":"7f37f127c93297148e7dbac392282653","Type":"ContainerStarted","Data":"f7724dbf552e0c1b93c8a5720ab5e7cf399f092817669e103288014e602e6eeb"}
Sep 18 13:58:34.287509 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.287489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal" event={"ID":"3e878fac6c20a59b3277411748de820b","Type":"ContainerStarted","Data":"cb6f28fc26b54c12160f15ed0af0eedc8b39f6dfa2a465350f2c027737006f72"}
Sep 18 13:58:34.292752 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-system-cni-dir\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.292895 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292767    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-socket-dir-parent\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.292895 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-device-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.292895 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-modprobe-d\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.292895 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06abc05c-5806-4cd2-b1a8-0386e2c960cd-tmp\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.292895 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.292895 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfltk\" (UniqueName: \"kubernetes.io/projected/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-kube-api-access-xfltk\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.293204 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-kubelet\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.293204 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.293204 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292946    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.293204 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.293204 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.292990    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-sys\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.293204 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293014    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-cni-bin\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.293204 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5e28ca0-b379-4dca-8f03-affaffaafade-ovnkube-config\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.293204 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94143e9a-35af-4e2f-92fe-e27c520db0ab-tmp-dir\") pod \"node-resolver-plgx7\" (UID: \"94143e9a-35af-4e2f-92fe-e27c520db0ab\") " pod="openshift-dns/node-resolver-plgx7"
Sep 18 13:58:34.293204 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9srqp\" (UniqueName: \"kubernetes.io/projected/b00ac136-4ce2-4f92-85cb-ffa07460a93c-kube-api-access-9srqp\") pod \"iptables-alerter-945p8\" (UID: \"b00ac136-4ce2-4f92-85cb-ffa07460a93c\") " pod="openshift-network-operator/iptables-alerter-945p8"
Sep 18 13:58:34.293204 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-hostroot\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.293204 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k48g6\" (UniqueName: \"kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6\") pod \"network-check-target-j5xvn\" (UID: \"9a53946c-99aa-4bb6-83f6-57a08c271b16\") " pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da7590f0-0553-4140-a8dd-57e3f3b254cf-cni-binary-copy\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-kubernetes\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-sysctl-conf\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-systemd\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-log-socket\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5e28ca0-b379-4dca-8f03-affaffaafade-env-overrides\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94143e9a-35af-4e2f-92fe-e27c520db0ab-tmp-dir\") pod \"node-resolver-plgx7\" (UID: \"94143e9a-35af-4e2f-92fe-e27c520db0ab\") " pod="openshift-dns/node-resolver-plgx7"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-sysconfig\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-run\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6812878a-b037-40b6-89c1-445d49376f65-kubelet-config\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94143e9a-35af-4e2f-92fe-e27c520db0ab-hosts-file\") pod \"node-resolver-plgx7\" (UID: \"94143e9a-35af-4e2f-92fe-e27c520db0ab\") " pod="openshift-dns/node-resolver-plgx7"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-systemd-units\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-slash\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/94143e9a-35af-4e2f-92fe-e27c520db0ab-hosts-file\") pod \"node-resolver-plgx7\" (UID: \"94143e9a-35af-4e2f-92fe-e27c520db0ab\") " pod="openshift-dns/node-resolver-plgx7"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.293571    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwkd\" (UniqueName: \"kubernetes.io/projected/8bfae7fb-425a-493c-927c-877fd442c666-kube-api-access-vtwkd\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:34.293669 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6812878a-b037-40b6-89c1-445d49376f65-kubelet-config\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.293641    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs podName:8bfae7fb-425a-493c-927c-877fd442c666 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:34.79362384 +0000 UTC m=+2.950144112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs") pod "network-metrics-daemon-bzklt" (UID: "8bfae7fb-425a-493c-927c-877fd442c666") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b00ac136-4ce2-4f92-85cb-ffa07460a93c-host-slash\") pod \"iptables-alerter-945p8\" (UID: \"b00ac136-4ce2-4f92-85cb-ffa07460a93c\") " pod="openshift-network-operator/iptables-alerter-945p8"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-tuned\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-etc-openvswitch\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-node-log\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svdql\" (UniqueName: \"kubernetes.io/projected/94143e9a-35af-4e2f-92fe-e27c520db0ab-kube-api-access-svdql\") pod \"node-resolver-plgx7\" (UID: \"94143e9a-35af-4e2f-92fe-e27c520db0ab\") " pod="openshift-dns/node-resolver-plgx7"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b00ac136-4ce2-4f92-85cb-ffa07460a93c-host-slash\") pod \"iptables-alerter-945p8\" (UID: \"b00ac136-4ce2-4f92-85cb-ffa07460a93c\") " pod="openshift-network-operator/iptables-alerter-945p8"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-run-netns\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-registration-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.293962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-sys-fs\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-cni-netd\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/336c2a92-68fa-4de6-ba21-fbeab8836c37-serviceca\") pod \"node-ca-98pdk\" (UID: \"336c2a92-68fa-4de6-ba21-fbeab8836c37\") " pod="openshift-image-registry/node-ca-98pdk"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b00ac136-4ce2-4f92-85cb-ffa07460a93c-iptables-alerter-script\") pod \"iptables-alerter-945p8\" (UID: \"b00ac136-4ce2-4f92-85cb-ffa07460a93c\") " pod="openshift-network-operator/iptables-alerter-945p8"
Sep 18 13:58:34.294405 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-var-lib-kubelet\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-socket-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-host\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-cnibin\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-run-systemd\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17faa55e-6e23-473c-828c-8727a86c9977-agent-certs\") pod \"konnectivity-agent-wtgbx\" (UID: \"17faa55e-6e23-473c-828c-8727a86c9977\") " pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294308    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-cni-dir\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-cnibin\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-run-k8s-cni-cncf-io\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2wh8\" (UniqueName: \"kubernetes.io/projected/281ee328-b978-47ba-9368-14bbb3b50505-kube-api-access-b2wh8\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-var-lib-openvswitch\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/336c2a92-68fa-4de6-ba21-fbeab8836c37-serviceca\") pod \"node-ca-98pdk\" (UID: \"336c2a92-68fa-4de6-ba21-fbeab8836c37\") " pod="openshift-image-registry/node-ca-98pdk"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-run-openvswitch\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-var-lib-cni-bin\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-run-netns\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:34.295090 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-etc-kubernetes\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.294620    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-var-lib-kubelet\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b00ac136-4ce2-4f92-85cb-ffa07460a93c-iptables-alerter-script\") pod \"iptables-alerter-945p8\" (UID: \"b00ac136-4ce2-4f92-85cb-ffa07460a93c\") " pod="openshift-network-operator/iptables-alerter-945p8"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.294673    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret podName:6812878a-b037-40b6-89c1-445d49376f65 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:34.794657627 +0000 UTC m=+2.951177891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret") pod "global-pull-secret-syncer-pp5sf" (UID: "6812878a-b037-40b6-89c1-445d49376f65") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-system-cni-dir\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-run-ovn\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6812878a-b037-40b6-89c1-445d49376f65-dbus\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17faa55e-6e23-473c-828c-8727a86c9977-konnectivity-ca\") pod \"konnectivity-agent-wtgbx\" (UID: \"17faa55e-6e23-473c-828c-8727a86c9977\") " pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/336c2a92-68fa-4de6-ba21-fbeab8836c37-host\") pod \"node-ca-98pdk\" (UID: \"336c2a92-68fa-4de6-ba21-fbeab8836c37\") " pod="openshift-image-registry/node-ca-98pdk"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6812878a-b037-40b6-89c1-445d49376f65-dbus\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-run-multus-certs\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfsn\" (UniqueName: \"kubernetes.io/projected/06abc05c-5806-4cd2-b1a8-0386e2c960cd-kube-api-access-mdfsn\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pppzd\" (UniqueName: \"kubernetes.io/projected/336c2a92-68fa-4de6-ba21-fbeab8836c37-kube-api-access-pppzd\") pod \"node-ca-98pdk\" (UID: \"336c2a92-68fa-4de6-ba21-fbeab8836c37\") " pod="openshift-image-registry/node-ca-98pdk"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294937    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-sysctl-d\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-lib-modules\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.295781 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.294978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5e28ca0-b379-4dca-8f03-affaffaafade-ovn-node-metrics-cert\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.296411 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.295003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxv54\" (UniqueName: \"kubernetes.io/projected/e5e28ca0-b379-4dca-8f03-affaffaafade-kube-api-access-jxv54\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.296411 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.295056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-var-lib-cni-multus\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.296411 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.295080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-conf-dir\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.296411 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.295107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-daemon-config\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.296411 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.295182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx29b\" (UniqueName: \"kubernetes.io/projected/da7590f0-0553-4140-a8dd-57e3f3b254cf-kube-api-access-lx29b\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.296411 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.295184    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 13:58:34.296411 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.295209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.296411 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.295234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5e28ca0-b379-4dca-8f03-affaffaafade-ovnkube-script-lib\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.296411 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.295273    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/336c2a92-68fa-4de6-ba21-fbeab8836c37-host\") pod \"node-ca-98pdk\" (UID: \"336c2a92-68fa-4de6-ba21-fbeab8836c37\") " pod="openshift-image-registry/node-ca-98pdk"
Sep 18 13:58:34.296411 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.296188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/17faa55e-6e23-473c-828c-8727a86c9977-konnectivity-ca\") pod \"konnectivity-agent-wtgbx\" (UID: \"17faa55e-6e23-473c-828c-8727a86c9977\") " pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:34.299945 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.299928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/17faa55e-6e23-473c-828c-8727a86c9977-agent-certs\") pod \"konnectivity-agent-wtgbx\" (UID: \"17faa55e-6e23-473c-828c-8727a86c9977\") " pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:34.308499 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.308481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svdql\" (UniqueName: \"kubernetes.io/projected/94143e9a-35af-4e2f-92fe-e27c520db0ab-kube-api-access-svdql\") pod \"node-resolver-plgx7\" (UID: \"94143e9a-35af-4e2f-92fe-e27c520db0ab\") " pod="openshift-dns/node-resolver-plgx7"
Sep 18 13:58:34.308653 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.308635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9srqp\" (UniqueName: \"kubernetes.io/projected/b00ac136-4ce2-4f92-85cb-ffa07460a93c-kube-api-access-9srqp\") pod \"iptables-alerter-945p8\" (UID: \"b00ac136-4ce2-4f92-85cb-ffa07460a93c\") " pod="openshift-network-operator/iptables-alerter-945p8"
Sep 18 13:58:34.308733 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.308705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwkd\" (UniqueName: \"kubernetes.io/projected/8bfae7fb-425a-493c-927c-877fd442c666-kube-api-access-vtwkd\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:34.311287 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.311269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pppzd\" (UniqueName: \"kubernetes.io/projected/336c2a92-68fa-4de6-ba21-fbeab8836c37-kube-api-access-pppzd\") pod \"node-ca-98pdk\" (UID: \"336c2a92-68fa-4de6-ba21-fbeab8836c37\") " pod="openshift-image-registry/node-ca-98pdk"
Sep 18 13:58:34.395543 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-device-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.395653 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-modprobe-d\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.395653 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-device-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.395653 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06abc05c-5806-4cd2-b1a8-0386e2c960cd-tmp\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.395810 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.395810 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfltk\" (UniqueName: \"kubernetes.io/projected/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-kube-api-access-xfltk\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.395810 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-kubelet\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.395810 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.395810 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-sys\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395866    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-sys\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-cni-bin\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-kubelet\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5e28ca0-b379-4dca-8f03-affaffaafade-ovnkube-config\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.395989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396006    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-cni-bin\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-modprobe-d\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-hostroot\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.396081 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48g6\" (UniqueName: \"kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6\") pod \"network-check-target-j5xvn\" (UID: \"9a53946c-99aa-4bb6-83f6-57a08c271b16\") " pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da7590f0-0553-4140-a8dd-57e3f3b254cf-cni-binary-copy\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-hostroot\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-kubernetes\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-sysctl-conf\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-kubernetes\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-systemd\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-sysctl-conf\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-log-socket\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-systemd\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5e28ca0-b379-4dca-8f03-affaffaafade-env-overrides\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e5e28ca0-b379-4dca-8f03-affaffaafade-ovnkube-config\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-log-socket\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.396685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-sysconfig\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-run\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-sysconfig\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-systemd-units\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da7590f0-0553-4140-a8dd-57e3f3b254cf-cni-binary-copy\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-slash\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-run\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-tuned\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-systemd-units\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-slash\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-etc-openvswitch\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-node-log\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e5e28ca0-b379-4dca-8f03-affaffaafade-env-overrides\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-run-netns\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-etc-openvswitch\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-node-log\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.396997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-registration-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.397327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-run-netns\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-sys-fs\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-cni-netd\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-var-lib-kubelet\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-etc-selinux\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-cni-netd\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-registration-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-sys-fs\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-socket-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397241    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-var-lib-kubelet\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-host\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-cnibin\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-socket-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-run-systemd\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-cni-dir\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-host\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.398180 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-cnibin\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-run-systemd\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-run-k8s-cni-cncf-io\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-cnibin\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-cnibin\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2wh8\" (UniqueName: \"kubernetes.io/projected/281ee328-b978-47ba-9368-14bbb3b50505-kube-api-access-b2wh8\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-var-lib-openvswitch\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-cni-dir\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-run-k8s-cni-cncf-io\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/281ee328-b978-47ba-9368-14bbb3b50505-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-var-lib-openvswitch\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-run-openvswitch\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-var-lib-cni-bin\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397636    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-run-openvswitch\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-run-netns\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.398920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-var-lib-cni-bin\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-host-run-netns\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-etc-kubernetes\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-var-lib-kubelet\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-etc-kubernetes\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-system-cni-dir\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397805    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-system-cni-dir\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-run-ovn\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-run-multus-certs\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfsn\" (UniqueName: \"kubernetes.io/projected/06abc05c-5806-4cd2-b1a8-0386e2c960cd-kube-api-access-mdfsn\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-sysctl-d\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-lib-modules\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-run-multus-certs\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5e28ca0-b379-4dca-8f03-affaffaafade-ovn-node-metrics-cert\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxv54\" (UniqueName: \"kubernetes.io/projected/e5e28ca0-b379-4dca-8f03-affaffaafade-kube-api-access-jxv54\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-var-lib-cni-multus\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398085    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-conf-dir\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.399713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-daemon-config\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx29b\" (UniqueName: \"kubernetes.io/projected/da7590f0-0553-4140-a8dd-57e3f3b254cf-kube-api-access-lx29b\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5e28ca0-b379-4dca-8f03-affaffaafade-ovnkube-script-lib\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-system-cni-dir\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-socket-dir-parent\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-lib-modules\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-conf-dir\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-sysctl-d\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06abc05c-5806-4cd2-b1a8-0386e2c960cd-var-lib-kubelet\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.397855    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e5e28ca0-b379-4dca-8f03-affaffaafade-run-ovn\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-system-cni-dir\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-host-var-lib-cni-multus\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-socket-dir-parent\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.398930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e5e28ca0-b379-4dca-8f03-affaffaafade-ovnkube-script-lib\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.399457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06abc05c-5806-4cd2-b1a8-0386e2c960cd-tmp\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.399458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da7590f0-0553-4140-a8dd-57e3f3b254cf-multus-daemon-config\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.400479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.400194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06abc05c-5806-4cd2-b1a8-0386e2c960cd-etc-tuned\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.401500 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.401478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e5e28ca0-b379-4dca-8f03-affaffaafade-ovn-node-metrics-cert\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.422719 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.422698    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx29b\" (UniqueName: \"kubernetes.io/projected/da7590f0-0553-4140-a8dd-57e3f3b254cf-kube-api-access-lx29b\") pod \"multus-5w4nj\" (UID: \"da7590f0-0553-4140-a8dd-57e3f3b254cf\") " pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.422889 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.422871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxv54\" (UniqueName: \"kubernetes.io/projected/e5e28ca0-b379-4dca-8f03-affaffaafade-kube-api-access-jxv54\") pod \"ovnkube-node-pqnf9\" (UID: \"e5e28ca0-b379-4dca-8f03-affaffaafade\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.425597 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.425575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfltk\" (UniqueName: \"kubernetes.io/projected/f344d0dd-1f21-46d4-8c4e-4869c2a7f090-kube-api-access-xfltk\") pod \"multus-additional-cni-plugins-ttrvx\" (UID: \"f344d0dd-1f21-46d4-8c4e-4869c2a7f090\") " pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.425868 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.425848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfsn\" (UniqueName: \"kubernetes.io/projected/06abc05c-5806-4cd2-b1a8-0386e2c960cd-kube-api-access-mdfsn\") pod \"tuned-d7tbh\" (UID: \"06abc05c-5806-4cd2-b1a8-0386e2c960cd\") " pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.426576 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.426555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2wh8\" (UniqueName: \"kubernetes.io/projected/281ee328-b978-47ba-9368-14bbb3b50505-kube-api-access-b2wh8\") pod \"aws-ebs-csi-driver-node-6wspl\" (UID: \"281ee328-b978-47ba-9368-14bbb3b50505\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.429010 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.428981    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:58:34.429010 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.429006    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:58:34.429168 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.429018    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k48g6 for pod openshift-network-diagnostics/network-check-target-j5xvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:34.429168 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.429089    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6 podName:9a53946c-99aa-4bb6-83f6-57a08c271b16 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:34.929073438 +0000 UTC m=+3.085593701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k48g6" (UniqueName: "kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6") pod "network-check-target-j5xvn" (UID: "9a53946c-99aa-4bb6-83f6-57a08c271b16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:34.488975 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.488955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-plgx7"
Sep 18 13:58:34.497610 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.497584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-98pdk"
Sep 18 13:58:34.507657 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.507639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-945p8"
Sep 18 13:58:34.517333 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.517316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:34.524606 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.524585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5w4nj"
Sep 18 13:58:34.532259 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.532243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttrvx"
Sep 18 13:58:34.541599 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.541581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:34.550185 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.550170    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl"
Sep 18 13:58:34.554766 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.554749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d7tbh"
Sep 18 13:58:34.722361 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.722290    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:58:34.800561 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.800527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:34.800709 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:34.800575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:34.800709 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.800674    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:34.800709 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.800681    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:34.800836 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.800738    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs podName:8bfae7fb-425a-493c-927c-877fd442c666 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:35.800721674 +0000 UTC m=+3.957241936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs") pod "network-metrics-daemon-bzklt" (UID: "8bfae7fb-425a-493c-927c-877fd442c666") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:34.800836 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:34.800753    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret podName:6812878a-b037-40b6-89c1-445d49376f65 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:35.800747008 +0000 UTC m=+3.957267266 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret") pod "global-pull-secret-syncer-pp5sf" (UID: "6812878a-b037-40b6-89c1-445d49376f65") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:35.002811 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.002746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48g6\" (UniqueName: \"kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6\") pod \"network-check-target-j5xvn\" (UID: \"9a53946c-99aa-4bb6-83f6-57a08c271b16\") " pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:35.002947 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:35.002903    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:58:35.002947 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:35.002926    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:58:35.002947 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:35.002940    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k48g6 for pod openshift-network-diagnostics/network-check-target-j5xvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:35.003063 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:35.003002    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6 podName:9a53946c-99aa-4bb6-83f6-57a08c271b16 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:36.002984507 +0000 UTC m=+4.159504787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k48g6" (UniqueName: "kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6") pod "network-check-target-j5xvn" (UID: "9a53946c-99aa-4bb6-83f6-57a08c271b16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:35.248228 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:35.248200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281ee328_b978_47ba_9368_14bbb3b50505.slice/crio-ef5d37e6d4378461d775e8430bd5ef98b879d11a78bc86ed46d7942a396a4005 WatchSource:0}: Error finding container ef5d37e6d4378461d775e8430bd5ef98b879d11a78bc86ed46d7942a396a4005: Status 404 returned error can't find the container with id ef5d37e6d4378461d775e8430bd5ef98b879d11a78bc86ed46d7942a396a4005
Sep 18 13:58:35.249828 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:35.249807    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94143e9a_35af_4e2f_92fe_e27c520db0ab.slice/crio-5e65dafa3b5de11f59fe18413f3949521a85324159fc25e0bf40ca30dffcfd3a WatchSource:0}: Error finding container 5e65dafa3b5de11f59fe18413f3949521a85324159fc25e0bf40ca30dffcfd3a: Status 404 returned error can't find the container with id 5e65dafa3b5de11f59fe18413f3949521a85324159fc25e0bf40ca30dffcfd3a
Sep 18 13:58:35.250736 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:35.250680    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06abc05c_5806_4cd2_b1a8_0386e2c960cd.slice/crio-2cecd6328f656ea0186f36d1db6bb4e39d2d679b11ccabb156e72e812cee3c6c WatchSource:0}: Error finding container 2cecd6328f656ea0186f36d1db6bb4e39d2d679b11ccabb156e72e812cee3c6c: Status 404 returned error can't find the container with id 2cecd6328f656ea0186f36d1db6bb4e39d2d679b11ccabb156e72e812cee3c6c
Sep 18 13:58:35.252860 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:35.252732    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00ac136_4ce2_4f92_85cb_ffa07460a93c.slice/crio-17d1c7eec5398c7ba7317dd7bc4c70c81f5a1b3437a8d9ba1f318bef1a1a482e WatchSource:0}: Error finding container 17d1c7eec5398c7ba7317dd7bc4c70c81f5a1b3437a8d9ba1f318bef1a1a482e: Status 404 returned error can't find the container with id 17d1c7eec5398c7ba7317dd7bc4c70c81f5a1b3437a8d9ba1f318bef1a1a482e
Sep 18 13:58:35.253185 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.253118    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:58:35.267851 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.267819    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:53:34 +0000 UTC" deadline="2028-04-20 21:13:08.869121786 +0000 UTC"
Sep 18 13:58:35.267916 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.267850    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13927h14m33.601275949s"
Sep 18 13:58:35.274718 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:35.274700    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17faa55e_6e23_473c_828c_8727a86c9977.slice/crio-f4b42dcf0c35ca618842f9935f6aa194d0a82aa25f8c4f929ddd4551937eae7c WatchSource:0}: Error finding container f4b42dcf0c35ca618842f9935f6aa194d0a82aa25f8c4f929ddd4551937eae7c: Status 404 returned error can't find the container with id f4b42dcf0c35ca618842f9935f6aa194d0a82aa25f8c4f929ddd4551937eae7c
Sep 18 13:58:35.275781 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:35.275758    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e28ca0_b379_4dca_8f03_affaffaafade.slice/crio-f371a5f47a01efc0df09720b901c584b48e760b06f7ff1f783d429d94104337d WatchSource:0}: Error finding container f371a5f47a01efc0df09720b901c584b48e760b06f7ff1f783d429d94104337d: Status 404 returned error can't find the container with id f371a5f47a01efc0df09720b901c584b48e760b06f7ff1f783d429d94104337d
Sep 18 13:58:35.276561 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:35.276480    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336c2a92_68fa_4de6_ba21_fbeab8836c37.slice/crio-0b9cb7664c8c77545c9b3df233c245e5a3424d9b8f3f74062c7ce785277b4a11 WatchSource:0}: Error finding container 0b9cb7664c8c77545c9b3df233c245e5a3424d9b8f3f74062c7ce785277b4a11: Status 404 returned error can't find the container with id 0b9cb7664c8c77545c9b3df233c245e5a3424d9b8f3f74062c7ce785277b4a11
Sep 18 13:58:35.277906 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:35.277853    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf344d0dd_1f21_46d4_8c4e_4869c2a7f090.slice/crio-24962181c846bfa06d457675448322bfe1433dda338b49bd2ee488a2e5e0fe57 WatchSource:0}: Error finding container 24962181c846bfa06d457675448322bfe1433dda338b49bd2ee488a2e5e0fe57: Status 404 returned error can't find the container with id 24962181c846bfa06d457675448322bfe1433dda338b49bd2ee488a2e5e0fe57
Sep 18 13:58:35.278677 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:58:35.278661    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7590f0_0553_4140_a8dd_57e3f3b254cf.slice/crio-11fe8b5f428ffa1b899a06c197d3f47b224d2ff60df428875afecbc9323bff56 WatchSource:0}: Error finding container 11fe8b5f428ffa1b899a06c197d3f47b224d2ff60df428875afecbc9323bff56: Status 404 returned error can't find the container with id 11fe8b5f428ffa1b899a06c197d3f47b224d2ff60df428875afecbc9323bff56
Sep 18 13:58:35.290251 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.290224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-plgx7" event={"ID":"94143e9a-35af-4e2f-92fe-e27c520db0ab","Type":"ContainerStarted","Data":"5e65dafa3b5de11f59fe18413f3949521a85324159fc25e0bf40ca30dffcfd3a"}
Sep 18 13:58:35.292833 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.292801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl" event={"ID":"281ee328-b978-47ba-9368-14bbb3b50505","Type":"ContainerStarted","Data":"ef5d37e6d4378461d775e8430bd5ef98b879d11a78bc86ed46d7942a396a4005"}
Sep 18 13:58:35.293885 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.293866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9" event={"ID":"e5e28ca0-b379-4dca-8f03-affaffaafade","Type":"ContainerStarted","Data":"f371a5f47a01efc0df09720b901c584b48e760b06f7ff1f783d429d94104337d"}
Sep 18 13:58:35.294891 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.294870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-98pdk" event={"ID":"336c2a92-68fa-4de6-ba21-fbeab8836c37","Type":"ContainerStarted","Data":"0b9cb7664c8c77545c9b3df233c245e5a3424d9b8f3f74062c7ce785277b4a11"}
Sep 18 13:58:35.296246 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.296220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-945p8" event={"ID":"b00ac136-4ce2-4f92-85cb-ffa07460a93c","Type":"ContainerStarted","Data":"17d1c7eec5398c7ba7317dd7bc4c70c81f5a1b3437a8d9ba1f318bef1a1a482e"}
Sep 18 13:58:35.297827 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.297801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5w4nj" event={"ID":"da7590f0-0553-4140-a8dd-57e3f3b254cf","Type":"ContainerStarted","Data":"11fe8b5f428ffa1b899a06c197d3f47b224d2ff60df428875afecbc9323bff56"}
Sep 18 13:58:35.299299 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.299269    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrvx" event={"ID":"f344d0dd-1f21-46d4-8c4e-4869c2a7f090","Type":"ContainerStarted","Data":"24962181c846bfa06d457675448322bfe1433dda338b49bd2ee488a2e5e0fe57"}
Sep 18 13:58:35.300505 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.300479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wtgbx" event={"ID":"17faa55e-6e23-473c-828c-8727a86c9977","Type":"ContainerStarted","Data":"f4b42dcf0c35ca618842f9935f6aa194d0a82aa25f8c4f929ddd4551937eae7c"}
Sep 18 13:58:35.301653 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.301631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d7tbh" event={"ID":"06abc05c-5806-4cd2-b1a8-0386e2c960cd","Type":"ContainerStarted","Data":"2cecd6328f656ea0186f36d1db6bb4e39d2d679b11ccabb156e72e812cee3c6c"}
Sep 18 13:58:35.809588 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.809478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:35.809588 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:35.809566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:35.809803 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:35.809666    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:35.809803 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:35.809682    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:35.809803 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:35.809730    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs podName:8bfae7fb-425a-493c-927c-877fd442c666 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:37.809712935 +0000 UTC m=+5.966233210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs") pod "network-metrics-daemon-bzklt" (UID: "8bfae7fb-425a-493c-927c-877fd442c666") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:35.809803 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:35.809747    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret podName:6812878a-b037-40b6-89c1-445d49376f65 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:37.809739433 +0000 UTC m=+5.966259693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret") pod "global-pull-secret-syncer-pp5sf" (UID: "6812878a-b037-40b6-89c1-445d49376f65") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:36.011523 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:36.011490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48g6\" (UniqueName: \"kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6\") pod \"network-check-target-j5xvn\" (UID: \"9a53946c-99aa-4bb6-83f6-57a08c271b16\") " pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:36.011725 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:36.011708    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:58:36.011788 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:36.011737    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:58:36.011788 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:36.011749    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k48g6 for pod openshift-network-diagnostics/network-check-target-j5xvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:36.011889 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:36.011800    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6 podName:9a53946c-99aa-4bb6-83f6-57a08c271b16 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:38.011784035 +0000 UTC m=+6.168304307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k48g6" (UniqueName: "kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6") pod "network-check-target-j5xvn" (UID: "9a53946c-99aa-4bb6-83f6-57a08c271b16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:36.268978 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:36.268912    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:53:34 +0000 UTC" deadline="2028-05-11 05:53:58.304332433 +0000 UTC"
Sep 18 13:58:36.268978 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:36.268974    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14415h55m22.035361912s"
Sep 18 13:58:36.281546 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:36.281523    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:36.281674 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:36.281631    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:36.282062 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:36.282044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:36.282218 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:36.282156    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:36.282286 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:36.282252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:36.282437 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:36.282331    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:36.311362 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:36.311331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal" event={"ID":"3e878fac6c20a59b3277411748de820b","Type":"ContainerStarted","Data":"c2303b1b16aec6f829fe7e69512b25d5161bdd15bf10622c920ecca77e210ef6"}
Sep 18 13:58:36.321260 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:36.321234    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f37f127c93297148e7dbac392282653" containerID="06353d46bce8872528db9809d580b776cc1f9b94fb190810d2b3f958af46b025" exitCode=0
Sep 18 13:58:36.321368 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:36.321277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal" event={"ID":"7f37f127c93297148e7dbac392282653","Type":"ContainerDied","Data":"06353d46bce8872528db9809d580b776cc1f9b94fb190810d2b3f958af46b025"}
Sep 18 13:58:36.347613 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:36.347567    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-55.ec2.internal" podStartSLOduration=3.347551269 podStartE2EDuration="3.347551269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:36.329304743 +0000 UTC m=+4.485825025" watchObservedRunningTime="2026-09-18 13:58:36.347551269 +0000 UTC m=+4.504071551"
Sep 18 13:58:37.351565 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:37.351244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal" event={"ID":"7f37f127c93297148e7dbac392282653","Type":"ContainerStarted","Data":"118c983bb67fd4847bf69e5b2341462f27a8c1dae167715f7b0cf69b1ea53842"}
Sep 18 13:58:37.831874 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:37.831840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:37.832069 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:37.831905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:37.832069 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:37.832036    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:37.832220 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:37.832095    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret podName:6812878a-b037-40b6-89c1-445d49376f65 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:41.832077018 +0000 UTC m=+9.988597291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret") pod "global-pull-secret-syncer-pp5sf" (UID: "6812878a-b037-40b6-89c1-445d49376f65") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:37.832511 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:37.832488    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:37.832633 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:37.832545    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs podName:8bfae7fb-425a-493c-927c-877fd442c666 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:41.832530902 +0000 UTC m=+9.989051174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs") pod "network-metrics-daemon-bzklt" (UID: "8bfae7fb-425a-493c-927c-877fd442c666") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:38.033547 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:38.033223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48g6\" (UniqueName: \"kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6\") pod \"network-check-target-j5xvn\" (UID: \"9a53946c-99aa-4bb6-83f6-57a08c271b16\") " pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:38.033547 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:38.033383    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:58:38.033547 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:38.033404    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:58:38.033547 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:38.033416    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k48g6 for pod openshift-network-diagnostics/network-check-target-j5xvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:38.033547 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:38.033471    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6 podName:9a53946c-99aa-4bb6-83f6-57a08c271b16 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:42.033454333 +0000 UTC m=+10.189974612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k48g6" (UniqueName: "kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6") pod "network-check-target-j5xvn" (UID: "9a53946c-99aa-4bb6-83f6-57a08c271b16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:38.282555 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:38.281824    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:38.282555 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:38.281950    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:38.282555 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:38.282337    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:38.282555 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:38.282438    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:38.282555 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:38.281771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:38.282555 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:38.282524    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:40.281988 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:40.281680    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:40.281988 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:40.281799    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:40.282553 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:40.282049    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:40.282553 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:40.282184    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:40.282553 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:40.282286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:40.282553 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:40.282394    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:41.864890 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:41.864580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:41.864890 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:41.864644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:41.864890 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:41.864780    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:41.864890 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:41.864852    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret podName:6812878a-b037-40b6-89c1-445d49376f65 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:49.864832573 +0000 UTC m=+18.021352831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret") pod "global-pull-secret-syncer-pp5sf" (UID: "6812878a-b037-40b6-89c1-445d49376f65") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:41.864890 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:41.864780    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:41.865549 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:41.864947    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs podName:8bfae7fb-425a-493c-927c-877fd442c666 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:49.864923443 +0000 UTC m=+18.021443715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs") pod "network-metrics-daemon-bzklt" (UID: "8bfae7fb-425a-493c-927c-877fd442c666") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:42.066170 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:42.066092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48g6\" (UniqueName: \"kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6\") pod \"network-check-target-j5xvn\" (UID: \"9a53946c-99aa-4bb6-83f6-57a08c271b16\") " pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:42.066336 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:42.066272    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:58:42.066336 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:42.066294    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:58:42.066336 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:42.066307    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k48g6 for pod openshift-network-diagnostics/network-check-target-j5xvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:42.066489 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:42.066357    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6 podName:9a53946c-99aa-4bb6-83f6-57a08c271b16 nodeName:}" failed. No retries permitted until 2026-09-18 13:58:50.066342905 +0000 UTC m=+18.222863167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k48g6" (UniqueName: "kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6") pod "network-check-target-j5xvn" (UID: "9a53946c-99aa-4bb6-83f6-57a08c271b16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:42.283256 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:42.282820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:42.283256 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:42.282923    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:42.283256 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:42.283028    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:42.283256 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:42.283097    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:42.283256 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:42.283150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:42.283256 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:42.283212    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:44.284456 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:44.284034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:44.284456 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:44.284034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:44.284456 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:44.284152    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:44.284456 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:44.284150    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:44.284456 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:44.284216    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:44.284456 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:44.284307    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:46.284970 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:46.284943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:46.285410 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:46.285003    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:46.285410 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:46.285021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:46.285410 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:46.285108    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:46.285410 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:46.285192    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:46.285410 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:46.285268    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:48.282018 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:48.281983    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:48.282018 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:48.282022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:48.282517 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:48.281983    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:48.282517 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:48.282120    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:48.282517 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:48.282206    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:48.282517 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:48.282326    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:49.931466 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:49.931437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:49.931881 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:49.931506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:49.931881 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:49.931608    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:49.931881 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:49.931622    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:49.931881 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:49.931672    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret podName:6812878a-b037-40b6-89c1-445d49376f65 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:05.93165549 +0000 UTC m=+34.088175752 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret") pod "global-pull-secret-syncer-pp5sf" (UID: "6812878a-b037-40b6-89c1-445d49376f65") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:58:49.931881 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:49.931687    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs podName:8bfae7fb-425a-493c-927c-877fd442c666 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:05.931680217 +0000 UTC m=+34.088200475 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs") pod "network-metrics-daemon-bzklt" (UID: "8bfae7fb-425a-493c-927c-877fd442c666") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:58:50.133196 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:50.133165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48g6\" (UniqueName: \"kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6\") pod \"network-check-target-j5xvn\" (UID: \"9a53946c-99aa-4bb6-83f6-57a08c271b16\") " pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:50.133382 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:50.133344    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:58:50.133382 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:50.133367    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:58:50.133382 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:50.133378    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k48g6 for pod openshift-network-diagnostics/network-check-target-j5xvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:50.133523 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:50.133442    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6 podName:9a53946c-99aa-4bb6-83f6-57a08c271b16 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:06.133424514 +0000 UTC m=+34.289944785 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k48g6" (UniqueName: "kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6") pod "network-check-target-j5xvn" (UID: "9a53946c-99aa-4bb6-83f6-57a08c271b16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:58:50.282320 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:50.282212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:50.282320 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:50.282246    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:50.282320 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:50.282218    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:50.282585 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:50.282338    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:50.282585 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:50.282433    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:50.282585 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:50.282530    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:52.282601 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:52.282567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:52.283037 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:52.282640    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:52.283037 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:52.282732    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:52.283037 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:52.282832    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:52.283037 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:52.282884    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:52.283037 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:52.282938    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:53.383996 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.383756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d7tbh" event={"ID":"06abc05c-5806-4cd2-b1a8-0386e2c960cd","Type":"ContainerStarted","Data":"c3a1175f8f08a8651aed06a7ad976f8e28e0a0a7b1417058f3075ce6dab27cb9"}
Sep 18 13:58:53.385249 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.385229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-plgx7" event={"ID":"94143e9a-35af-4e2f-92fe-e27c520db0ab","Type":"ContainerStarted","Data":"39578817b6b6d70d0fb64a2a86279625769f03fa5d5d748818276b94df89e67b"}
Sep 18 13:58:53.386724 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.386698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl" event={"ID":"281ee328-b978-47ba-9368-14bbb3b50505","Type":"ContainerStarted","Data":"ccc2d772a655641e314abc2564a86d7ccc32e299013bfd6df87ddabc22ac94f4"}
Sep 18 13:58:53.389487 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.389465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9" event={"ID":"e5e28ca0-b379-4dca-8f03-affaffaafade","Type":"ContainerStarted","Data":"bb3762dda318958e60a17b3af27c4b59d37ba78a63a3e2beeefa2a1f0b5a3792"}
Sep 18 13:58:53.389578 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.389533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9" event={"ID":"e5e28ca0-b379-4dca-8f03-affaffaafade","Type":"ContainerStarted","Data":"89dc81acad755966b7520973f99ae9ecbb9cd45860487efa6e37d247cc0a2dae"}
Sep 18 13:58:53.389578 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.389574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9" event={"ID":"e5e28ca0-b379-4dca-8f03-affaffaafade","Type":"ContainerStarted","Data":"2aae85b62c11f2843edebf104b792423ccf38e4c7710e0f5dbcbc07a7947cd75"}
Sep 18 13:58:53.389650 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.389607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9" event={"ID":"e5e28ca0-b379-4dca-8f03-affaffaafade","Type":"ContainerStarted","Data":"7d52c5c1cb10f30cc5124149c91d3879f1415d334d972065b4d5b7a13b1406ed"}
Sep 18 13:58:53.389650 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.389634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9" event={"ID":"e5e28ca0-b379-4dca-8f03-affaffaafade","Type":"ContainerStarted","Data":"ff494b4e4ade9f710cb5bb604c80d47fe89042cd942993486ea064ea6a054465"}
Sep 18 13:58:53.389736 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.389659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9" event={"ID":"e5e28ca0-b379-4dca-8f03-affaffaafade","Type":"ContainerStarted","Data":"b43a483b760a468d1c4af8e9944757b575d67f3ba4c852503b84b0f5ea7a2b09"}
Sep 18 13:58:53.390847 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.390825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-98pdk" event={"ID":"336c2a92-68fa-4de6-ba21-fbeab8836c37","Type":"ContainerStarted","Data":"aa911e0fe46107ff4d590273525b6a461e58e258f8dc96cc6f6ee59b1d5823a6"}
Sep 18 13:58:53.392364 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.392339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5w4nj" event={"ID":"da7590f0-0553-4140-a8dd-57e3f3b254cf","Type":"ContainerStarted","Data":"9fdcf4bfae6cf4712082bfa9c6496538bb4fe578a1793ef6b92ad1f1b53b5892"}
Sep 18 13:58:53.393771 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.393750    2563 generic.go:356] "Generic (PLEG): container finished" podID="f344d0dd-1f21-46d4-8c4e-4869c2a7f090" containerID="45cc348666bc2214f6700bcb7cf914cc297909fc6134f4f8d596592891878d76" exitCode=0
Sep 18 13:58:53.393855 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.393830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrvx" event={"ID":"f344d0dd-1f21-46d4-8c4e-4869c2a7f090","Type":"ContainerDied","Data":"45cc348666bc2214f6700bcb7cf914cc297909fc6134f4f8d596592891878d76"}
Sep 18 13:58:53.395270 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.395251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wtgbx" event={"ID":"17faa55e-6e23-473c-828c-8727a86c9977","Type":"ContainerStarted","Data":"c56788f1d203d6266bf590dfebf6eceb7f66d8c4bc8406a0a0c239a84b7ef680"}
Sep 18 13:58:53.443618 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.443576    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d7tbh" podStartSLOduration=4.264365503 podStartE2EDuration="21.443565343s" totalImagesPullingTime="17.17919984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:53.431188764 +0000 UTC m=+21.587709044" watchObservedRunningTime="2026-09-18 13:58:53.443565343 +0000 UTC m=+21.600085623"
Sep 18 13:58:53.443803 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.443760    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-55.ec2.internal" podStartSLOduration=20.443747161 podStartE2EDuration="20.443747161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:37.378486582 +0000 UTC m=+5.535006864" watchObservedRunningTime="2026-09-18 13:58:53.443747161 +0000 UTC m=+21.600267440"
Sep 18 13:58:53.547537 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.547483    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-98pdk" podStartSLOduration=4.419248341 podStartE2EDuration="21.547462302s" totalImagesPullingTime="17.128213961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:53.472875301 +0000 UTC m=+21.629395582" watchObservedRunningTime="2026-09-18 13:58:53.547462302 +0000 UTC m=+21.703982583"
Sep 18 13:58:53.547845 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.547809    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wtgbx" podStartSLOduration=8.22209137 podStartE2EDuration="21.547769196s" totalImagesPullingTime="13.325677826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:53.546870266 +0000 UTC m=+21.703390545" watchObservedRunningTime="2026-09-18 13:58:53.547769196 +0000 UTC m=+21.704289476"
Sep 18 13:58:53.626783 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.626737    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5w4nj" podStartSLOduration=4.448409147 podStartE2EDuration="21.626721887s" totalImagesPullingTime="17.17831274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:53.624334033 +0000 UTC m=+21.780854313" watchObservedRunningTime="2026-09-18 13:58:53.626721887 +0000 UTC m=+21.783242166"
Sep 18 13:58:53.707932 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:53.707867    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-plgx7" podStartSLOduration=4.534828739 podStartE2EDuration="21.707845918s" totalImagesPullingTime="17.173017179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:53.706840897 +0000 UTC m=+21.863361180" watchObservedRunningTime="2026-09-18 13:58:53.707845918 +0000 UTC m=+21.864366199"
Sep 18 13:58:54.233087 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:54.233065    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 13:58:54.252754 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:54.252663    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T13:58:54.233082702Z","UUID":"b35327ad-9ea8-4dc8-99e8-402cfbe5702d","Handler":null,"Name":"","Endpoint":""}
Sep 18 13:58:54.254223 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:54.254194    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 13:58:54.254320 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:54.254229    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 13:58:54.281788 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:54.281760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:54.281946 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:54.281873    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:54.281946 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:54.281893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:54.281946 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:54.281772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:54.282079 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:54.281997    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:54.282079 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:54.282067    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:54.399646 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:54.399563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl" event={"ID":"281ee328-b978-47ba-9368-14bbb3b50505","Type":"ContainerStarted","Data":"d4773610e9fe2ab4bb3577d8a2346bb992f94b6154a993f1f8c93924c9371505"}
Sep 18 13:58:54.401230 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:54.401194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-945p8" event={"ID":"b00ac136-4ce2-4f92-85cb-ffa07460a93c","Type":"ContainerStarted","Data":"8ec6e783166a3a6ae81015ad5d089d7f2eabd552b82d40d6712914e0c0a87ee5"}
Sep 18 13:58:54.427393 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:54.427344    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-945p8" podStartSLOduration=5.248003303 podStartE2EDuration="22.427328439s" totalImagesPullingTime="17.179325136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:54.427271404 +0000 UTC m=+22.583791696" watchObservedRunningTime="2026-09-18 13:58:54.427328439 +0000 UTC m=+22.583848729"
Sep 18 13:58:55.408012 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:55.407753    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9" event={"ID":"e5e28ca0-b379-4dca-8f03-affaffaafade","Type":"ContainerStarted","Data":"279570a57ad6325a126050861bac139947afc8f48b7ddc02fc8d86fa73336ef8"}
Sep 18 13:58:55.570821 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:55.570747    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:55.571322 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:55.571299    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:56.281982 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:56.281948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:56.281982 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:56.281959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:56.282318 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:56.281948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:56.282318 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:56.282073    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:56.282318 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:56.282162    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:56.282318 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:56.282229    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:56.415106 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:56.415068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl" event={"ID":"281ee328-b978-47ba-9368-14bbb3b50505","Type":"ContainerStarted","Data":"d7c4eb7aee76ab9dbf9e78b1d89290fe0d5d7d838d289409825a8bc560c476a0"}
Sep 18 13:58:56.415542 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:56.415392    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:56.415871 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:56.415850    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wtgbx"
Sep 18 13:58:56.462201 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:56.462161    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6wspl" podStartSLOduration=4.298276018 podStartE2EDuration="24.462148453s" totalImagesPullingTime="20.163872435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:56.441481333 +0000 UTC m=+24.598001612" watchObservedRunningTime="2026-09-18 13:58:56.462148453 +0000 UTC m=+24.618668735"
Sep 18 13:58:58.282676 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:58.282286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:58.282676 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:58.282286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:58.282676 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:58.282429    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:58.282676 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:58.282435    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:58.282676 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:58.282540    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:58:58.282676 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:58.282627    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:58.421863 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:58.421836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrvx" event={"ID":"f344d0dd-1f21-46d4-8c4e-4869c2a7f090","Type":"ContainerStarted","Data":"6cddbc6fbfb724b41b8a652f16ae22c844d153be6e9c5257c174251df88a72f4"}
Sep 18 13:58:58.425684 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:58.425659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9" event={"ID":"e5e28ca0-b379-4dca-8f03-affaffaafade","Type":"ContainerStarted","Data":"1c0d44634cf9ff433473485a41d4c94d0d509819c14256f294bb8d49f1a95ea2"}
Sep 18 13:58:58.475784 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:58.475745    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9" podStartSLOduration=9.187638511 podStartE2EDuration="26.475730855s" totalImagesPullingTime="17.288092344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:58.475538665 +0000 UTC m=+26.632058928" watchObservedRunningTime="2026-09-18 13:58:58.475730855 +0000 UTC m=+26.632251134"
Sep 18 13:58:59.429845 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.429663    2563 generic.go:356] "Generic (PLEG): container finished" podID="f344d0dd-1f21-46d4-8c4e-4869c2a7f090" containerID="6cddbc6fbfb724b41b8a652f16ae22c844d153be6e9c5257c174251df88a72f4" exitCode=0
Sep 18 13:58:59.430378 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.429741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrvx" event={"ID":"f344d0dd-1f21-46d4-8c4e-4869c2a7f090","Type":"ContainerDied","Data":"6cddbc6fbfb724b41b8a652f16ae22c844d153be6e9c5257c174251df88a72f4"}
Sep 18 13:58:59.430980 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.430484    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:59.430980 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.430559    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:59.430980 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.430610    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:59.445310 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.445291    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:59.445467 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.445451    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:58:59.975047 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.975016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5xvn"]
Sep 18 13:58:59.975226 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.975120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:58:59.975286 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:59.975218    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:58:59.978755 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.978728    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bzklt"]
Sep 18 13:58:59.978894 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.978838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:58:59.978971 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:59.978950    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:58:59.980517 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.980494    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pp5sf"]
Sep 18 13:58:59.980618 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:58:59.980579    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:58:59.980664 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:58:59.980644    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:59:01.281861 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:01.281833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:59:01.282266 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:01.281957    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:59:01.282266 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:01.281992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:59:01.282266 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:01.282082    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:59:01.437265 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:01.437239    2563 generic.go:356] "Generic (PLEG): container finished" podID="f344d0dd-1f21-46d4-8c4e-4869c2a7f090" containerID="ac2087106cd968f1fd9c87bade602540b45d111608a9129f7bfbbf6ba92ed48d" exitCode=0
Sep 18 13:59:01.437423 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:01.437311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrvx" event={"ID":"f344d0dd-1f21-46d4-8c4e-4869c2a7f090","Type":"ContainerDied","Data":"ac2087106cd968f1fd9c87bade602540b45d111608a9129f7bfbbf6ba92ed48d"}
Sep 18 13:59:02.282964 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:02.282941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:59:02.283411 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:02.283038    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:59:03.282185 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:03.282006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:59:03.282318 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:03.282006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:59:03.282318 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:03.282273    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:59:03.282396 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:03.282322    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:59:03.444482 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:03.444453    2563 generic.go:356] "Generic (PLEG): container finished" podID="f344d0dd-1f21-46d4-8c4e-4869c2a7f090" containerID="825420344b8ff6f0e14b4d08fecdf56c672fbb62d3599ec1cc7fc55964efe357" exitCode=0
Sep 18 13:59:03.444907 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:03.444492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrvx" event={"ID":"f344d0dd-1f21-46d4-8c4e-4869c2a7f090","Type":"ContainerDied","Data":"825420344b8ff6f0e14b4d08fecdf56c672fbb62d3599ec1cc7fc55964efe357"}
Sep 18 13:59:04.284851 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:04.284827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:59:04.284990 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:04.284954    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp5sf" podUID="6812878a-b037-40b6-89c1-445d49376f65"
Sep 18 13:59:05.282096 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:05.282067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:59:05.282096 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:05.282083    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:59:05.282655 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:05.282234    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bzklt" podUID="8bfae7fb-425a-493c-927c-877fd442c666"
Sep 18 13:59:05.282655 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:05.282307    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5xvn" podUID="9a53946c-99aa-4bb6-83f6-57a08c271b16"
Sep 18 13:59:05.957957 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:05.957924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:59:05.958113 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:05.957964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:59:05.958113 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:05.958048    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:59:05.958113 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:05.958055    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:59:05.958113 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:05.958092    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret podName:6812878a-b037-40b6-89c1-445d49376f65 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:37.958080878 +0000 UTC m=+66.114601149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret") pod "global-pull-secret-syncer-pp5sf" (UID: "6812878a-b037-40b6-89c1-445d49376f65") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:59:05.958113 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:05.958105    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs podName:8bfae7fb-425a-493c-927c-877fd442c666 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:37.958099518 +0000 UTC m=+66.114619776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs") pod "network-metrics-daemon-bzklt" (UID: "8bfae7fb-425a-493c-927c-877fd442c666") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:59:06.124196 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.124162    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-55.ec2.internal" event="NodeReady"
Sep 18 13:59:06.124344 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.124299    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 13:59:06.159810 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.159779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48g6\" (UniqueName: \"kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6\") pod \"network-check-target-j5xvn\" (UID: \"9a53946c-99aa-4bb6-83f6-57a08c271b16\") " pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:59:06.159960 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.159941    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:59:06.160009 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.159966    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:59:06.160009 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.159980    2563 projected.go:196] Error preparing data for projected volume kube-api-access-k48g6 for pod openshift-network-diagnostics/network-check-target-j5xvn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:59:06.160074 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.160038    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6 podName:9a53946c-99aa-4bb6-83f6-57a08c271b16 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:38.160018493 +0000 UTC m=+66.316538753 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k48g6" (UniqueName: "kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6") pod "network-check-target-j5xvn" (UID: "9a53946c-99aa-4bb6-83f6-57a08c271b16") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:59:06.162579 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.162555    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-tj226"]
Sep 18 13:59:06.164727 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.164708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.166846 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.166819    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"]
Sep 18 13:59:06.168988 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.168967    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 13:59:06.169084 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.169025    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 13:59:06.169084 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.169040    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 13:59:06.169271 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.169114    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 13:59:06.169271 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.169207    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pjgct\""
Sep 18 13:59:06.170412 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.170394    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-764b645cd6-x9x9r"]
Sep 18 13:59:06.170643 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.170620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:06.173270 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.173245    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 13:59:06.174236 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.173987    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-758d69797b-d24nl"]
Sep 18 13:59:06.174236 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.174024    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 13:59:06.174394 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.174266    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 13:59:06.174394 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.174305    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.174394 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.174371    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cf2zn\""
Sep 18 13:59:06.174539 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.174417    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 13:59:06.175722 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.175704    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 13:59:06.176899 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.176881    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 13:59:06.177777 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.177758    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.178864 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.178845    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 13:59:06.179047 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.179029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mtthk\""
Sep 18 13:59:06.179972 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.179951    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 13:59:06.180539 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.180517    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 13:59:06.180638 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.180576    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 13:59:06.180638 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.180604    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 13:59:06.181540 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.181519    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w5qd9\""
Sep 18 13:59:06.183939 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.183922    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 13:59:06.184185 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.184095    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 13:59:06.184185 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.184005    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 13:59:06.184868 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.184836    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-tj226"]
Sep 18 13:59:06.185537 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.185406    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k9kwv"]
Sep 18 13:59:06.187343 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.187219    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 13:59:06.190342 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.190321    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"]
Sep 18 13:59:06.190443 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.190412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.193589 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.193566    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-758d69797b-d24nl"]
Sep 18 13:59:06.194704 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.194685    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bjl8s"]
Sep 18 13:59:06.195576 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.195556    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 13:59:06.195951 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.195933    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fh88p\""
Sep 18 13:59:06.196244 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.196228    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 13:59:06.198637 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.198619    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k9kwv"]
Sep 18 13:59:06.198785 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.198720    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bjl8s"
Sep 18 13:59:06.201502 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.201441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5l9f5\""
Sep 18 13:59:06.202058 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.202039    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 13:59:06.203907 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.203885    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 13:59:06.204061 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.204046    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 13:59:06.206337 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.206317    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-764b645cd6-x9x9r"]
Sep 18 13:59:06.213441 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.213419    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bjl8s"]
Sep 18 13:59:06.260525 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.260495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f9fd0-371e-4d59-a868-42ba681905e7-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.260525 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.260525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxnm\" (UniqueName: \"kubernetes.io/projected/be1f9fd0-371e-4d59-a868-42ba681905e7-kube-api-access-9xxnm\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.260764 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.260547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e8a3b04-7240-4c24-a4c2-133a57d91595-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:06.260764 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.260615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/be1f9fd0-371e-4d59-a868-42ba681905e7-snapshots\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.260764 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.260658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be1f9fd0-371e-4d59-a868-42ba681905e7-serving-cert\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.260764 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.260682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f9fd0-371e-4d59-a868-42ba681905e7-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.260764 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.260701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:06.260764 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.260725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvz5\" (UniqueName: \"kubernetes.io/projected/1e8a3b04-7240-4c24-a4c2-133a57d91595-kube-api-access-6lvz5\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:06.260764 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.260749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be1f9fd0-371e-4d59-a868-42ba681905e7-tmp\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.282457 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.282432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:59:06.284898 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.284684    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:59:06.360386 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.360350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wtgbx_17faa55e-6e23-473c-828c-8727a86c9977/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:06.361239 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f9fd0-371e-4d59-a868-42ba681905e7-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.361353 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:06.361353 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvz5\" (UniqueName: \"kubernetes.io/projected/1e8a3b04-7240-4c24-a4c2-133a57d91595-kube-api-access-6lvz5\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:06.361353 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fvc4\" (UniqueName: \"kubernetes.io/projected/66650627-a6b5-4d1f-b5c6-3f812de684eb-kube-api-access-9fvc4\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.361353 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmsx\" (UniqueName: \"kubernetes.io/projected/d0d7ff07-d797-44c6-8c50-9312493190c1-kube-api-access-wdmsx\") pod \"ingress-canary-bjl8s\" (UID: \"d0d7ff07-d797-44c6-8c50-9312493190c1\") " pod="openshift-ingress-canary/ingress-canary-bjl8s"
Sep 18 13:59:06.361353 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-stats-auth\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.361600 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.361362    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:06.361600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-installation-pull-secrets\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.361600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be1f9fd0-371e-4d59-a868-42ba681905e7-tmp\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.361600 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.361418    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls podName:1e8a3b04-7240-4c24-a4c2-133a57d91595 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:06.861401363 +0000 UTC m=+35.017921636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-fc4t7" (UID: "1e8a3b04-7240-4c24-a4c2-133a57d91595") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:06.361600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66650627-a6b5-4d1f-b5c6-3f812de684eb-config-volume\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.361600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f9fd0-371e-4d59-a868-42ba681905e7-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.361600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-default-certificate\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.361600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361561    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxnm\" (UniqueName: \"kubernetes.io/projected/be1f9fd0-371e-4d59-a868-42ba681905e7-kube-api-access-9xxnm\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.361600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm5v7\" (UniqueName: \"kubernetes.io/projected/2bc70927-d434-49f3-b09a-dda6af701e30-kube-api-access-hm5v7\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.361955 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjzm\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-kube-api-access-kkjzm\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.361955 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e8a3b04-7240-4c24-a4c2-133a57d91595-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:06.361955 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-certificates\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.361955 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0d7ff07-d797-44c6-8c50-9312493190c1-cert\") pod \"ingress-canary-bjl8s\" (UID: \"d0d7ff07-d797-44c6-8c50-9312493190c1\") " pod="openshift-ingress-canary/ingress-canary-bjl8s"
Sep 18 13:59:06.361955 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be1f9fd0-371e-4d59-a868-42ba681905e7-tmp\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.361955 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/be1f9fd0-371e-4d59-a868-42ba681905e7-snapshots\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.361955 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.361812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f9fd0-371e-4d59-a868-42ba681905e7-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.362686 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e8a3b04-7240-4c24-a4c2-133a57d91595-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:06.362686 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.362686 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.362686 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be1f9fd0-371e-4d59-a868-42ba681905e7-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.362686 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-image-registry-private-configuration\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.362686 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.362686 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-bound-sa-token\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.362686 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/be1f9fd0-371e-4d59-a868-42ba681905e7-snapshots\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.363097 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66650627-a6b5-4d1f-b5c6-3f812de684eb-tmp-dir\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.363097 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.363097 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-ca-trust-extracted\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.363097 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-trusted-ca\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.363097 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.362870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be1f9fd0-371e-4d59-a868-42ba681905e7-serving-cert\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.367121 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.367098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be1f9fd0-371e-4d59-a868-42ba681905e7-serving-cert\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.370278 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.370253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxnm\" (UniqueName: \"kubernetes.io/projected/be1f9fd0-371e-4d59-a868-42ba681905e7-kube-api-access-9xxnm\") pod \"insights-operator-6f9c644dd9-tj226\" (UID: \"be1f9fd0-371e-4d59-a868-42ba681905e7\") " pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.370768 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.370743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvz5\" (UniqueName: \"kubernetes.io/projected/1e8a3b04-7240-4c24-a4c2-133a57d91595-kube-api-access-6lvz5\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:06.463429 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.463429 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.463429 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-image-registry-private-configuration\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.463610 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.463610 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.463489    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:59:06.463610 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.463538    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:59:06.463610 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.463547    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:06.963528454 +0000 UTC m=+35.120048716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : configmap references non-existent config key: service-ca.crt
Sep 18 13:59:06.463610 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.463550    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758d69797b-d24nl: secret "image-registry-tls" not found
Sep 18 13:59:06.463610 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.463594    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls podName:ee72d2ae-3ac8-418d-b0a6-3a45e0121d60 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:06.963583556 +0000 UTC m=+35.120103841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls") pod "image-registry-758d69797b-d24nl" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60") : secret "image-registry-tls" not found
Sep 18 13:59:06.463841 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.463615    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls podName:66650627-a6b5-4d1f-b5c6-3f812de684eb nodeName:}" failed. No retries permitted until 2026-09-18 13:59:06.963606832 +0000 UTC m=+35.120127107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls") pod "dns-default-k9kwv" (UID: "66650627-a6b5-4d1f-b5c6-3f812de684eb") : secret "dns-default-metrics-tls" not found
Sep 18 13:59:06.463841 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-bound-sa-token\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.463841 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66650627-a6b5-4d1f-b5c6-3f812de684eb-tmp-dir\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.463841 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.463841 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-ca-trust-extracted\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.463841 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-trusted-ca\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.464114 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvc4\" (UniqueName: \"kubernetes.io/projected/66650627-a6b5-4d1f-b5c6-3f812de684eb-kube-api-access-9fvc4\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.464114 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmsx\" (UniqueName: \"kubernetes.io/projected/d0d7ff07-d797-44c6-8c50-9312493190c1-kube-api-access-wdmsx\") pod \"ingress-canary-bjl8s\" (UID: \"d0d7ff07-d797-44c6-8c50-9312493190c1\") " pod="openshift-ingress-canary/ingress-canary-bjl8s"
Sep 18 13:59:06.464114 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-stats-auth\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.464114 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-installation-pull-secrets\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.464114 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66650627-a6b5-4d1f-b5c6-3f812de684eb-config-volume\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.464114 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.463979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66650627-a6b5-4d1f-b5c6-3f812de684eb-tmp-dir\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.464114 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.464030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-default-certificate\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.464114 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.464062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm5v7\" (UniqueName: \"kubernetes.io/projected/2bc70927-d434-49f3-b09a-dda6af701e30-kube-api-access-hm5v7\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.464114 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.464086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjzm\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-kube-api-access-kkjzm\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.464541 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.464150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-certificates\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.464541 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.464174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0d7ff07-d797-44c6-8c50-9312493190c1-cert\") pod \"ingress-canary-bjl8s\" (UID: \"d0d7ff07-d797-44c6-8c50-9312493190c1\") " pod="openshift-ingress-canary/ingress-canary-bjl8s"
Sep 18 13:59:06.464541 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.464229    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:59:06.464541 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.464286    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:06.964264973 +0000 UTC m=+35.120785235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : secret "router-metrics-certs-default" not found
Sep 18 13:59:06.464541 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.464298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-ca-trust-extracted\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.464541 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.464518    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66650627-a6b5-4d1f-b5c6-3f812de684eb-config-volume\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.465613 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.465570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-certificates\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.465960 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.465916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-trusted-ca\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.467755 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.467673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-image-registry-private-configuration\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.467755 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.467705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0d7ff07-d797-44c6-8c50-9312493190c1-cert\") pod \"ingress-canary-bjl8s\" (UID: \"d0d7ff07-d797-44c6-8c50-9312493190c1\") " pod="openshift-ingress-canary/ingress-canary-bjl8s"
Sep 18 13:59:06.467755 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.467721    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wtgbx_17faa55e-6e23-473c-828c-8727a86c9977/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:06.467755 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.467748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-stats-auth\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.467995 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.467924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-default-certificate\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.471882 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.471860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-installation-pull-secrets\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.479273 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.479252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmsx\" (UniqueName: \"kubernetes.io/projected/d0d7ff07-d797-44c6-8c50-9312493190c1-kube-api-access-wdmsx\") pod \"ingress-canary-bjl8s\" (UID: \"d0d7ff07-d797-44c6-8c50-9312493190c1\") " pod="openshift-ingress-canary/ingress-canary-bjl8s"
Sep 18 13:59:06.479413 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.479310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvc4\" (UniqueName: \"kubernetes.io/projected/66650627-a6b5-4d1f-b5c6-3f812de684eb-kube-api-access-9fvc4\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.479715 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.479686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjzm\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-kube-api-access-kkjzm\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.480443 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.480422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-tj226"
Sep 18 13:59:06.481396 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.481378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm5v7\" (UniqueName: \"kubernetes.io/projected/2bc70927-d434-49f3-b09a-dda6af701e30-kube-api-access-hm5v7\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.481479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.481451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-bound-sa-token\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.519494 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.519475    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bjl8s"
Sep 18 13:59:06.665107 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.665071    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9wlm8"]
Sep 18 13:59:06.668227 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.668201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9wlm8"
Sep 18 13:59:06.671087 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.671062    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:59:06.671227 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.671195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-52ms6\""
Sep 18 13:59:06.671227 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.671063    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 13:59:06.687276 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.687257    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9wlm8"]
Sep 18 13:59:06.766482 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.766413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4vg\" (UniqueName: \"kubernetes.io/projected/2e656651-372f-4cc7-9778-ba467cb45310-kube-api-access-pf4vg\") pod \"volume-data-source-validator-5c4db8bdf9-9wlm8\" (UID: \"2e656651-372f-4cc7-9778-ba467cb45310\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9wlm8"
Sep 18 13:59:06.777273 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.777243    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"]
Sep 18 13:59:06.780559 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.780542    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:06.786076 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.785837    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:59:06.786076 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.785869    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 13:59:06.786076 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.785919    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 13:59:06.786076 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.785943    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4ks8d\""
Sep 18 13:59:06.795185 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.795166    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"]
Sep 18 13:59:06.867836 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.867812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:06.867980 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.867908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4vg\" (UniqueName: \"kubernetes.io/projected/2e656651-372f-4cc7-9778-ba467cb45310-kube-api-access-pf4vg\") pod \"volume-data-source-validator-5c4db8bdf9-9wlm8\" (UID: \"2e656651-372f-4cc7-9778-ba467cb45310\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9wlm8"
Sep 18 13:59:06.867980 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.867964    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:06.868086 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.868039    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls podName:1e8a3b04-7240-4c24-a4c2-133a57d91595 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:07.868015514 +0000 UTC m=+36.024535791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-fc4t7" (UID: "1e8a3b04-7240-4c24-a4c2-133a57d91595") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:06.878563 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.878541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4vg\" (UniqueName: \"kubernetes.io/projected/2e656651-372f-4cc7-9778-ba467cb45310-kube-api-access-pf4vg\") pod \"volume-data-source-validator-5c4db8bdf9-9wlm8\" (UID: \"2e656651-372f-4cc7-9778-ba467cb45310\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9wlm8"
Sep 18 13:59:06.969083 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.969056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.969243 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.969087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:06.969243 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.969111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:06.969357 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.969245    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:59:06.969357 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.969254    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:07.969233327 +0000 UTC m=+36.125753610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : configmap references non-existent config key: service-ca.crt
Sep 18 13:59:06.969357 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.969251    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:59:06.969357 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.969285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:06.969357 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.969264    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758d69797b-d24nl: secret "image-registry-tls" not found
Sep 18 13:59:06.969357 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.969308    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:07.969291713 +0000 UTC m=+36.125811971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : secret "router-metrics-certs-default" not found
Sep 18 13:59:06.969357 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.969343    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls podName:ee72d2ae-3ac8-418d-b0a6-3a45e0121d60 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:07.969330831 +0000 UTC m=+36.125851104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls") pod "image-registry-758d69797b-d24nl" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60") : secret "image-registry-tls" not found
Sep 18 13:59:06.969657 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.969407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:06.969657 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.969451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x4fb\" (UniqueName: \"kubernetes.io/projected/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-kube-api-access-6x4fb\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:06.969657 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.969521    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:59:06.969657 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:06.969566    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls podName:66650627-a6b5-4d1f-b5c6-3f812de684eb nodeName:}" failed. No retries permitted until 2026-09-18 13:59:07.969553685 +0000 UTC m=+36.126073946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls") pod "dns-default-k9kwv" (UID: "66650627-a6b5-4d1f-b5c6-3f812de684eb") : secret "dns-default-metrics-tls" not found
Sep 18 13:59:06.977868 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:06.977849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9wlm8"
Sep 18 13:59:07.070638 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.070574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x4fb\" (UniqueName: \"kubernetes.io/projected/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-kube-api-access-6x4fb\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:07.070754 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.070677    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:07.070850 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.070830    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:59:07.070930 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.070892    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls podName:5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:07.570874717 +0000 UTC m=+35.727395019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8hchs" (UID: "5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250") : secret "samples-operator-tls" not found
Sep 18 13:59:07.079310 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.079279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x4fb\" (UniqueName: \"kubernetes.io/projected/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-kube-api-access-6x4fb\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:07.281841 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.281799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:59:07.281841 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.281823    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:59:07.284365 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.284343    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:59:07.284918 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.284375    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z6j5p\""
Sep 18 13:59:07.284918 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.284380    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 13:59:07.284918 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.284384    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 13:59:07.284918 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.284678    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8cnh9\""
Sep 18 13:59:07.574024 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.573992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:07.574214 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.574177    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:59:07.574264 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.574254    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls podName:5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:08.574235298 +0000 UTC m=+36.730755573 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8hchs" (UID: "5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250") : secret "samples-operator-tls" not found
Sep 18 13:59:07.659913 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.659884    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-v2rtv"]
Sep 18 13:59:07.663401 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.663376    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.665646 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.665620    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 13:59:07.666086 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.666034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 13:59:07.666203 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.666044    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 13:59:07.666259 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.666210    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pmln5\""
Sep 18 13:59:07.666259 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.666214    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:59:07.670636 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.670613    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 13:59:07.674042 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.674023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-v2rtv"]
Sep 18 13:59:07.775327 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.775295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-config\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.775432 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.775370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvwjj\" (UniqueName: \"kubernetes.io/projected/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-kube-api-access-kvwjj\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.775432 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.775405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-trusted-ca\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.775560 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.775445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-serving-cert\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.876188 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.876164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-config\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.876334 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.876196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:07.876334 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.876221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvwjj\" (UniqueName: \"kubernetes.io/projected/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-kube-api-access-kvwjj\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.876334 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.876248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-trusted-ca\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.876334 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.876283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-serving-cert\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.876334 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.876326    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:07.877493 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.877210    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls podName:1e8a3b04-7240-4c24-a4c2-133a57d91595 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:09.877186654 +0000 UTC m=+38.033706912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-fc4t7" (UID: "1e8a3b04-7240-4c24-a4c2-133a57d91595") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:07.880192 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.877673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-config\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.881790 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.881757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-trusted-ca\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.883599 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.883576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-serving-cert\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.885398 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.885380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvwjj\" (UniqueName: \"kubernetes.io/projected/d72a0a56-0157-4b87-8ddf-34754f6bdaa3-kube-api-access-kvwjj\") pod \"console-operator-fd4bb46c-v2rtv\" (UID: \"d72a0a56-0157-4b87-8ddf-34754f6bdaa3\") " pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.972021 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.971996    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9wlm8"]
Sep 18 13:59:07.975070 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.975043    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:07.975194 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.975106    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bjl8s"]
Sep 18 13:59:07.976011 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.975946    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-tj226"]
Sep 18 13:59:07.978903 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.978857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:07.978998 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.978968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:07.979054 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.979006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:07.979054 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:07.979033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:07.979186 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.979113    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:59:07.979245 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.979207    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls podName:66650627-a6b5-4d1f-b5c6-3f812de684eb nodeName:}" failed. No retries permitted until 2026-09-18 13:59:09.979183873 +0000 UTC m=+38.135704147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls") pod "dns-default-k9kwv" (UID: "66650627-a6b5-4d1f-b5c6-3f812de684eb") : secret "dns-default-metrics-tls" not found
Sep 18 13:59:07.979245 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.979223    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:59:07.979358 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.979275    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:09.979259963 +0000 UTC m=+38.135780225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : secret "router-metrics-certs-default" not found
Sep 18 13:59:07.979358 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.979349    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:09.979332127 +0000 UTC m=+38.135852405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : configmap references non-existent config key: service-ca.crt
Sep 18 13:59:07.979358 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.979349    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:59:07.979503 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.979369    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758d69797b-d24nl: secret "image-registry-tls" not found
Sep 18 13:59:07.979503 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:07.979409    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls podName:ee72d2ae-3ac8-418d-b0a6-3a45e0121d60 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:09.979397773 +0000 UTC m=+38.135918032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls") pod "image-registry-758d69797b-d24nl" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60") : secret "image-registry-tls" not found
Sep 18 13:59:08.003172 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:08.003144    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d7ff07_d797_44c6_8c50_9312493190c1.slice/crio-b938f70175d50a6951df3148053f3a91bbe82fa788ca440445d2e078c0c7a02c WatchSource:0}: Error finding container b938f70175d50a6951df3148053f3a91bbe82fa788ca440445d2e078c0c7a02c: Status 404 returned error can't find the container with id b938f70175d50a6951df3148053f3a91bbe82fa788ca440445d2e078c0c7a02c
Sep 18 13:59:08.003404 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:08.003381    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1f9fd0_371e_4d59_a868_42ba681905e7.slice/crio-24a1637fbbc9fa7c5f22ee5254d871acbad976961ef0cdff021ed28553cc488a WatchSource:0}: Error finding container 24a1637fbbc9fa7c5f22ee5254d871acbad976961ef0cdff021ed28553cc488a: Status 404 returned error can't find the container with id 24a1637fbbc9fa7c5f22ee5254d871acbad976961ef0cdff021ed28553cc488a
Sep 18 13:59:08.003612 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:08.003594    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e656651_372f_4cc7_9778_ba467cb45310.slice/crio-1e0b5ccbc3b3864713c80149afb47bb60c47591300131089adb5bf93652c1786 WatchSource:0}: Error finding container 1e0b5ccbc3b3864713c80149afb47bb60c47591300131089adb5bf93652c1786: Status 404 returned error can't find the container with id 1e0b5ccbc3b3864713c80149afb47bb60c47591300131089adb5bf93652c1786
Sep 18 13:59:08.252556 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:08.252525    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-v2rtv"]
Sep 18 13:59:08.255834 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:08.255805    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72a0a56_0157_4b87_8ddf_34754f6bdaa3.slice/crio-e93574ef67089533b392aec3c009a936bf3892b41261bf1db44b5daa2e18395c WatchSource:0}: Error finding container e93574ef67089533b392aec3c009a936bf3892b41261bf1db44b5daa2e18395c: Status 404 returned error can't find the container with id e93574ef67089533b392aec3c009a936bf3892b41261bf1db44b5daa2e18395c
Sep 18 13:59:08.456226 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:08.456194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tj226" event={"ID":"be1f9fd0-371e-4d59-a868-42ba681905e7","Type":"ContainerStarted","Data":"24a1637fbbc9fa7c5f22ee5254d871acbad976961ef0cdff021ed28553cc488a"}
Sep 18 13:59:08.457306 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:08.457275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bjl8s" event={"ID":"d0d7ff07-d797-44c6-8c50-9312493190c1","Type":"ContainerStarted","Data":"b938f70175d50a6951df3148053f3a91bbe82fa788ca440445d2e078c0c7a02c"}
Sep 18 13:59:08.460286 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:08.460255    2563 generic.go:356] "Generic (PLEG): container finished" podID="f344d0dd-1f21-46d4-8c4e-4869c2a7f090" containerID="524a5f4e8641e8e1d45e85f22178033ee56953d6a14bdc52f255e593353be1cb" exitCode=0
Sep 18 13:59:08.460390 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:08.460336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrvx" event={"ID":"f344d0dd-1f21-46d4-8c4e-4869c2a7f090","Type":"ContainerDied","Data":"524a5f4e8641e8e1d45e85f22178033ee56953d6a14bdc52f255e593353be1cb"}
Sep 18 13:59:08.461698 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:08.461673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9wlm8" event={"ID":"2e656651-372f-4cc7-9778-ba467cb45310","Type":"ContainerStarted","Data":"1e0b5ccbc3b3864713c80149afb47bb60c47591300131089adb5bf93652c1786"}
Sep 18 13:59:08.462992 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:08.462974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" event={"ID":"d72a0a56-0157-4b87-8ddf-34754f6bdaa3","Type":"ContainerStarted","Data":"e93574ef67089533b392aec3c009a936bf3892b41261bf1db44b5daa2e18395c"}
Sep 18 13:59:08.585947 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:08.585234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:08.585947 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:08.585774    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:59:08.585947 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:08.585839    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls podName:5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:10.58581934 +0000 UTC m=+38.742339626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8hchs" (UID: "5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250") : secret "samples-operator-tls" not found
Sep 18 13:59:09.470524 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:09.470413    2563 generic.go:356] "Generic (PLEG): container finished" podID="f344d0dd-1f21-46d4-8c4e-4869c2a7f090" containerID="17d8306ba7536b8f6c725c99e5d1f53d27f89be42004c68c81587a34573c7994" exitCode=0
Sep 18 13:59:09.470524 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:09.470467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrvx" event={"ID":"f344d0dd-1f21-46d4-8c4e-4869c2a7f090","Type":"ContainerDied","Data":"17d8306ba7536b8f6c725c99e5d1f53d27f89be42004c68c81587a34573c7994"}
Sep 18 13:59:09.898310 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:09.898231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:09.898442 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:09.898403    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:09.898499 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:09.898485    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls podName:1e8a3b04-7240-4c24-a4c2-133a57d91595 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:13.898462416 +0000 UTC m=+42.054982679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-fc4t7" (UID: "1e8a3b04-7240-4c24-a4c2-133a57d91595") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:09.919763 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:09.919732    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wtgbx_17faa55e-6e23-473c-828c-8727a86c9977/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:09.998904 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:09.998878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:09.999038 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:09.998967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:09.999038 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:09.998995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:09.999038 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:09.999019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:09.999191 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:09.999040    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:59:09.999191 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:09.999097    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls podName:66650627-a6b5-4d1f-b5c6-3f812de684eb nodeName:}" failed. No retries permitted until 2026-09-18 13:59:13.999080394 +0000 UTC m=+42.155600664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls") pod "dns-default-k9kwv" (UID: "66650627-a6b5-4d1f-b5c6-3f812de684eb") : secret "dns-default-metrics-tls" not found
Sep 18 13:59:09.999191 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:09.999158    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:13.999139233 +0000 UTC m=+42.155659511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : configmap references non-existent config key: service-ca.crt
Sep 18 13:59:09.999191 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:09.999171    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:59:09.999358 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:09.999209    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:13.999197134 +0000 UTC m=+42.155717417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : secret "router-metrics-certs-default" not found
Sep 18 13:59:09.999358 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:09.999252    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:59:09.999358 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:09.999266    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758d69797b-d24nl: secret "image-registry-tls" not found
Sep 18 13:59:09.999358 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:09.999312    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls podName:ee72d2ae-3ac8-418d-b0a6-3a45e0121d60 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:13.99929873 +0000 UTC m=+42.155819017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls") pod "image-registry-758d69797b-d24nl" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60") : secret "image-registry-tls" not found
Sep 18 13:59:10.026114 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:10.026082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wtgbx_17faa55e-6e23-473c-828c-8727a86c9977/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:10.604287 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:10.604209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:10.604707 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:10.604365    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:59:10.604707 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:10.604435    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls podName:5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:14.60441503 +0000 UTC m=+42.760935302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8hchs" (UID: "5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250") : secret "samples-operator-tls" not found
Sep 18 13:59:11.139832 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.139793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wtgbx_17faa55e-6e23-473c-828c-8727a86c9977/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:11.254070 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.254033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wtgbx_17faa55e-6e23-473c-828c-8727a86c9977/konnectivity-agent/0.log" limit=1024
Sep 18 13:59:11.665498 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.665460    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"]
Sep 18 13:59:11.689368 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.689347    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"]
Sep 18 13:59:11.689490 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.689425    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"
Sep 18 13:59:11.699339 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.699319    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 13:59:11.702570 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.702557    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 13:59:11.702834 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.702819    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 13:59:11.703347 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.703333    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ww7sm\""
Sep 18 13:59:11.703424 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.703336    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:59:11.710735 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.710717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j56zq\" (UniqueName: \"kubernetes.io/projected/aad65a15-66da-43c3-98e9-d46a8a109144-kube-api-access-j56zq\") pod \"kube-storage-version-migrator-operator-55889f7b47-jslm4\" (UID: \"aad65a15-66da-43c3-98e9-d46a8a109144\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"
Sep 18 13:59:11.710814 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.710761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aad65a15-66da-43c3-98e9-d46a8a109144-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jslm4\" (UID: \"aad65a15-66da-43c3-98e9-d46a8a109144\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"
Sep 18 13:59:11.710883 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.710817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aad65a15-66da-43c3-98e9-d46a8a109144-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jslm4\" (UID: \"aad65a15-66da-43c3-98e9-d46a8a109144\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"
Sep 18 13:59:11.811743 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.811717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j56zq\" (UniqueName: \"kubernetes.io/projected/aad65a15-66da-43c3-98e9-d46a8a109144-kube-api-access-j56zq\") pod \"kube-storage-version-migrator-operator-55889f7b47-jslm4\" (UID: \"aad65a15-66da-43c3-98e9-d46a8a109144\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"
Sep 18 13:59:11.811831 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.811754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aad65a15-66da-43c3-98e9-d46a8a109144-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jslm4\" (UID: \"aad65a15-66da-43c3-98e9-d46a8a109144\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"
Sep 18 13:59:11.811831 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.811777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aad65a15-66da-43c3-98e9-d46a8a109144-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jslm4\" (UID: \"aad65a15-66da-43c3-98e9-d46a8a109144\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"
Sep 18 13:59:11.812297 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.812280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aad65a15-66da-43c3-98e9-d46a8a109144-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-jslm4\" (UID: \"aad65a15-66da-43c3-98e9-d46a8a109144\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"
Sep 18 13:59:11.814774 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.814753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aad65a15-66da-43c3-98e9-d46a8a109144-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-jslm4\" (UID: \"aad65a15-66da-43c3-98e9-d46a8a109144\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"
Sep 18 13:59:11.824972 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.824953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j56zq\" (UniqueName: \"kubernetes.io/projected/aad65a15-66da-43c3-98e9-d46a8a109144-kube-api-access-j56zq\") pod \"kube-storage-version-migrator-operator-55889f7b47-jslm4\" (UID: \"aad65a15-66da-43c3-98e9-d46a8a109144\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"
Sep 18 13:59:11.997881 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:11.997859    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"
Sep 18 13:59:12.397754 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.397723    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4"]
Sep 18 13:59:12.401582 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:12.401553    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad65a15_66da_43c3_98e9_d46a8a109144.slice/crio-e3b89ff2d34c7503e4e6e76e8224dc80650504f67612f4d6c6f87f6f10433bed WatchSource:0}: Error finding container e3b89ff2d34c7503e4e6e76e8224dc80650504f67612f4d6c6f87f6f10433bed: Status 404 returned error can't find the container with id e3b89ff2d34c7503e4e6e76e8224dc80650504f67612f4d6c6f87f6f10433bed
Sep 18 13:59:12.480580 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.480540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9wlm8" event={"ID":"2e656651-372f-4cc7-9778-ba467cb45310","Type":"ContainerStarted","Data":"add6803bbf2d45dcb99800dfe6fec842b9152023984a1e0f78842757d0cccaeb"}
Sep 18 13:59:12.481790 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.481758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4" event={"ID":"aad65a15-66da-43c3-98e9-d46a8a109144","Type":"ContainerStarted","Data":"e3b89ff2d34c7503e4e6e76e8224dc80650504f67612f4d6c6f87f6f10433bed"}
Sep 18 13:59:12.483350 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.483334    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/0.log"
Sep 18 13:59:12.483445 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.483364    2563 generic.go:356] "Generic (PLEG): container finished" podID="d72a0a56-0157-4b87-8ddf-34754f6bdaa3" containerID="5a3e6ab9df2dc660dec35ab103ae145a42249f81559e2c010fd4ab7e0da380aa" exitCode=255
Sep 18 13:59:12.483445 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.483391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" event={"ID":"d72a0a56-0157-4b87-8ddf-34754f6bdaa3","Type":"ContainerDied","Data":"5a3e6ab9df2dc660dec35ab103ae145a42249f81559e2c010fd4ab7e0da380aa"}
Sep 18 13:59:12.483648 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.483631    2563 scope.go:117] "RemoveContainer" containerID="5a3e6ab9df2dc660dec35ab103ae145a42249f81559e2c010fd4ab7e0da380aa"
Sep 18 13:59:12.485043 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.485019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tj226" event={"ID":"be1f9fd0-371e-4d59-a868-42ba681905e7","Type":"ContainerStarted","Data":"bc8a124a1c411aabd24f955534faf9ad6a9c67db7af62c8f5830a793f1e1ee1c"}
Sep 18 13:59:12.486624 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.486602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bjl8s" event={"ID":"d0d7ff07-d797-44c6-8c50-9312493190c1","Type":"ContainerStarted","Data":"026cdca58cdddcbd8fb5394b726d0f76bb66528147fbe18349fe21f7352a61d9"}
Sep 18 13:59:12.490122 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.490104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttrvx" event={"ID":"f344d0dd-1f21-46d4-8c4e-4869c2a7f090","Type":"ContainerStarted","Data":"93dd5d95bd043eff1a8c849d7a5118707c68a67dbdad4e81fcb8e1f707657212"}
Sep 18 13:59:12.553701 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.553655    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-9wlm8" podStartSLOduration=2.875256723 podStartE2EDuration="6.553644425s" totalImagesPullingTime="3.678387702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:12.51030371 +0000 UTC m=+40.666823990" watchObservedRunningTime="2026-09-18 13:59:12.553644425 +0000 UTC m=+40.710164713"
Sep 18 13:59:12.586004 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.585957    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ttrvx" podStartSLOduration=11.526257947 podStartE2EDuration="40.58594551s" totalImagesPullingTime="29.059687563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:12.554457387 +0000 UTC m=+40.710977667" watchObservedRunningTime="2026-09-18 13:59:12.58594551 +0000 UTC m=+40.742465791"
Sep 18 13:59:12.586349 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.586321    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-tj226" podStartSLOduration=3.471994485 podStartE2EDuration="7.586314952s" totalImagesPullingTime="4.114320467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:12.585203276 +0000 UTC m=+40.741723557" watchObservedRunningTime="2026-09-18 13:59:12.586314952 +0000 UTC m=+40.742835226"
Sep 18 13:59:12.620002 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:12.619953    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bjl8s" podStartSLOduration=2.500879195 podStartE2EDuration="6.619936453s" totalImagesPullingTime="4.119057258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:12.619235629 +0000 UTC m=+40.775755915" watchObservedRunningTime="2026-09-18 13:59:12.619936453 +0000 UTC m=+40.776456735"
Sep 18 13:59:13.494800 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.494771    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/1.log"
Sep 18 13:59:13.495306 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.495285    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/0.log"
Sep 18 13:59:13.495376 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.495334    2563 generic.go:356] "Generic (PLEG): container finished" podID="d72a0a56-0157-4b87-8ddf-34754f6bdaa3" containerID="a278ebc8704ca690aa04f954db26042dfad021b33f6f3b951344f8f4c1936777" exitCode=255
Sep 18 13:59:13.495453 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.495435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" event={"ID":"d72a0a56-0157-4b87-8ddf-34754f6bdaa3","Type":"ContainerDied","Data":"a278ebc8704ca690aa04f954db26042dfad021b33f6f3b951344f8f4c1936777"}
Sep 18 13:59:13.495619 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.495538    2563 scope.go:117] "RemoveContainer" containerID="5a3e6ab9df2dc660dec35ab103ae145a42249f81559e2c010fd4ab7e0da380aa"
Sep 18 13:59:13.495734 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.495627    2563 scope.go:117] "RemoveContainer" containerID="a278ebc8704ca690aa04f954db26042dfad021b33f6f3b951344f8f4c1936777"
Sep 18 13:59:13.495843 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:13.495804    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-v2rtv_openshift-console-operator(d72a0a56-0157-4b87-8ddf-34754f6bdaa3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" podUID="d72a0a56-0157-4b87-8ddf-34754f6bdaa3"
Sep 18 13:59:13.680505 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.680382    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"]
Sep 18 13:59:13.711268 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.711199    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"
Sep 18 13:59:13.715857 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.715832    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 13:59:13.716030 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.715874    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 13:59:13.716030 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.715957    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dzp9z\""
Sep 18 13:59:13.717107 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.717050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"]
Sep 18 13:59:13.719506 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.719466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 13:59:13.719506 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.719469    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:59:13.728598 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.728579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/329d7a0f-ec4f-4aad-842c-7b31e1fd69d4-config\") pod \"service-ca-operator-57cdc4b768-pmhhb\" (UID: \"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"
Sep 18 13:59:13.728729 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.728621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/329d7a0f-ec4f-4aad-842c-7b31e1fd69d4-serving-cert\") pod \"service-ca-operator-57cdc4b768-pmhhb\" (UID: \"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"
Sep 18 13:59:13.728729 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.728710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwrtg\" (UniqueName: \"kubernetes.io/projected/329d7a0f-ec4f-4aad-842c-7b31e1fd69d4-kube-api-access-lwrtg\") pod \"service-ca-operator-57cdc4b768-pmhhb\" (UID: \"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"
Sep 18 13:59:13.752582 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.752533    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2vl5w"]
Sep 18 13:59:13.771062 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.771010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2vl5w"]
Sep 18 13:59:13.771184 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.771120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.776901 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.776884    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 13:59:13.777522 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.777505    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 13:59:13.777606 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.777563    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p5mbv\""
Sep 18 13:59:13.829189 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.829164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.829287 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.829213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/77cd4cd8-dd31-4511-a6ec-1f162d963772-crio-socket\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.829287 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.829236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxkbx\" (UniqueName: \"kubernetes.io/projected/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-api-access-lxkbx\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.829357 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.829297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/329d7a0f-ec4f-4aad-842c-7b31e1fd69d4-config\") pod \"service-ca-operator-57cdc4b768-pmhhb\" (UID: \"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"
Sep 18 13:59:13.829357 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.829319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/329d7a0f-ec4f-4aad-842c-7b31e1fd69d4-serving-cert\") pod \"service-ca-operator-57cdc4b768-pmhhb\" (UID: \"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"
Sep 18 13:59:13.829357 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.829342    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/77cd4cd8-dd31-4511-a6ec-1f162d963772-data-volume\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.829465 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.829384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwrtg\" (UniqueName: \"kubernetes.io/projected/329d7a0f-ec4f-4aad-842c-7b31e1fd69d4-kube-api-access-lwrtg\") pod \"service-ca-operator-57cdc4b768-pmhhb\" (UID: \"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"
Sep 18 13:59:13.829465 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.829415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.829932 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.829912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/329d7a0f-ec4f-4aad-842c-7b31e1fd69d4-config\") pod \"service-ca-operator-57cdc4b768-pmhhb\" (UID: \"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"
Sep 18 13:59:13.831648 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.831625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/329d7a0f-ec4f-4aad-842c-7b31e1fd69d4-serving-cert\") pod \"service-ca-operator-57cdc4b768-pmhhb\" (UID: \"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"
Sep 18 13:59:13.840035 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.840013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwrtg\" (UniqueName: \"kubernetes.io/projected/329d7a0f-ec4f-4aad-842c-7b31e1fd69d4-kube-api-access-lwrtg\") pod \"service-ca-operator-57cdc4b768-pmhhb\" (UID: \"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"
Sep 18 13:59:13.930607 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.930583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/77cd4cd8-dd31-4511-a6ec-1f162d963772-crio-socket\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.930713 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.930616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxkbx\" (UniqueName: \"kubernetes.io/projected/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-api-access-lxkbx\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.930774 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.930708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/77cd4cd8-dd31-4511-a6ec-1f162d963772-crio-socket\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.930831 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.930794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/77cd4cd8-dd31-4511-a6ec-1f162d963772-data-volume\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.930920 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.930904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.930976 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.930961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:13.931022 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.931002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.931072 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:13.931026    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:59:13.931119 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:13.931091    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm podName:77cd4cd8-dd31-4511-a6ec-1f162d963772 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:14.431073319 +0000 UTC m=+42.587593579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vl5w" (UID: "77cd4cd8-dd31-4511-a6ec-1f162d963772") : configmap "kube-rbac-proxy" not found
Sep 18 13:59:13.931119 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:13.931106    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:13.931254 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:13.931112    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:59:13.931254 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:13.931183    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls podName:1e8a3b04-7240-4c24-a4c2-133a57d91595 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:21.931166777 +0000 UTC m=+50.087687049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-fc4t7" (UID: "1e8a3b04-7240-4c24-a4c2-133a57d91595") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:13.931254 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:13.931204    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls podName:77cd4cd8-dd31-4511-a6ec-1f162d963772 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:14.431194004 +0000 UTC m=+42.587714264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vl5w" (UID: "77cd4cd8-dd31-4511-a6ec-1f162d963772") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:59:13.931372 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.931267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/77cd4cd8-dd31-4511-a6ec-1f162d963772-data-volume\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:13.943598 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:13.943578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxkbx\" (UniqueName: \"kubernetes.io/projected/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-api-access-lxkbx\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:14.022009 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.021959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"
Sep 18 13:59:14.031357 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.031334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:14.031452 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.031406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:14.031452 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.031426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:14.031452 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.031443    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:14.031609 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.031479    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:59:14.031609 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.031537    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls podName:66650627-a6b5-4d1f-b5c6-3f812de684eb nodeName:}" failed. No retries permitted until 2026-09-18 13:59:22.031512777 +0000 UTC m=+50.188033051 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls") pod "dns-default-k9kwv" (UID: "66650627-a6b5-4d1f-b5c6-3f812de684eb") : secret "dns-default-metrics-tls" not found
Sep 18 13:59:14.031609 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.031553    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:59:14.031609 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.031560    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:22.03154633 +0000 UTC m=+50.188066588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : configmap references non-existent config key: service-ca.crt
Sep 18 13:59:14.031609 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.031604    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:22.031591919 +0000 UTC m=+50.188112178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : secret "router-metrics-certs-default" not found
Sep 18 13:59:14.031609 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.031606    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:59:14.031864 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.031620    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758d69797b-d24nl: secret "image-registry-tls" not found
Sep 18 13:59:14.031864 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.031656    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls podName:ee72d2ae-3ac8-418d-b0a6-3a45e0121d60 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:22.031645245 +0000 UTC m=+50.188165505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls") pod "image-registry-758d69797b-d24nl" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60") : secret "image-registry-tls" not found
Sep 18 13:59:14.227296 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.227259    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-dsbvp"]
Sep 18 13:59:14.248295 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.248266    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-dsbvp"]
Sep 18 13:59:14.248432 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.248380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dsbvp"
Sep 18 13:59:14.255306 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.255284    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-g8nr4\""
Sep 18 13:59:14.334163 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.334050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26tn7\" (UniqueName: \"kubernetes.io/projected/ab11dd37-11a9-4a40-b00a-267f7080eb18-kube-api-access-26tn7\") pod \"network-check-source-7cbc45fddc-dsbvp\" (UID: \"ab11dd37-11a9-4a40-b00a-267f7080eb18\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dsbvp"
Sep 18 13:59:14.334284 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.334182    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb"]
Sep 18 13:59:14.435516 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.435488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:14.435682 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.435647    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:59:14.435682 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.435673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26tn7\" (UniqueName: \"kubernetes.io/projected/ab11dd37-11a9-4a40-b00a-267f7080eb18-kube-api-access-26tn7\") pod \"network-check-source-7cbc45fddc-dsbvp\" (UID: \"ab11dd37-11a9-4a40-b00a-267f7080eb18\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dsbvp"
Sep 18 13:59:14.435827 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.435735    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls podName:77cd4cd8-dd31-4511-a6ec-1f162d963772 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:15.435713429 +0000 UTC m=+43.592233701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vl5w" (UID: "77cd4cd8-dd31-4511-a6ec-1f162d963772") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:59:14.435827 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.435810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:14.435931 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.435901    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:59:14.435981 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.435969    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm podName:77cd4cd8-dd31-4511-a6ec-1f162d963772 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:15.435953034 +0000 UTC m=+43.592473292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vl5w" (UID: "77cd4cd8-dd31-4511-a6ec-1f162d963772") : configmap "kube-rbac-proxy" not found
Sep 18 13:59:14.447229 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.447207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26tn7\" (UniqueName: \"kubernetes.io/projected/ab11dd37-11a9-4a40-b00a-267f7080eb18-kube-api-access-26tn7\") pod \"network-check-source-7cbc45fddc-dsbvp\" (UID: \"ab11dd37-11a9-4a40-b00a-267f7080eb18\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dsbvp"
Sep 18 13:59:14.499953 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.499925    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/1.log"
Sep 18 13:59:14.500353 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.500328    2563 scope.go:117] "RemoveContainer" containerID="a278ebc8704ca690aa04f954db26042dfad021b33f6f3b951344f8f4c1936777"
Sep 18 13:59:14.500541 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.500501    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-v2rtv_openshift-console-operator(d72a0a56-0157-4b87-8ddf-34754f6bdaa3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" podUID="d72a0a56-0157-4b87-8ddf-34754f6bdaa3"
Sep 18 13:59:14.559408 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.559385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dsbvp"
Sep 18 13:59:14.600254 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:14.600232    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329d7a0f_ec4f_4aad_842c_7b31e1fd69d4.slice/crio-c5ea5d2ec855e82efb315c363b88fd9d1cf8a00d2ae916537ed4966d646c1d93 WatchSource:0}: Error finding container c5ea5d2ec855e82efb315c363b88fd9d1cf8a00d2ae916537ed4966d646c1d93: Status 404 returned error can't find the container with id c5ea5d2ec855e82efb315c363b88fd9d1cf8a00d2ae916537ed4966d646c1d93
Sep 18 13:59:14.638029 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.637884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:14.638151 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.638117    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:59:14.638211 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:14.638199    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls podName:5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:22.638179064 +0000 UTC m=+50.794699342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8hchs" (UID: "5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250") : secret "samples-operator-tls" not found
Sep 18 13:59:14.864645 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.864623    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-dsbvp"]
Sep 18 13:59:14.870846 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:14.870828    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-plgx7_94143e9a-35af-4e2f-92fe-e27c520db0ab/dns-node-resolver/0.log"
Sep 18 13:59:14.953309 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:14.953280    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab11dd37_11a9_4a40_b00a_267f7080eb18.slice/crio-6176b52c9733422804b51c296be84b1691dbe9223821bf8a36694dac2f4b88c6 WatchSource:0}: Error finding container 6176b52c9733422804b51c296be84b1691dbe9223821bf8a36694dac2f4b88c6: Status 404 returned error can't find the container with id 6176b52c9733422804b51c296be84b1691dbe9223821bf8a36694dac2f4b88c6
Sep 18 13:59:15.444976 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:15.444938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:15.445169 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:15.445016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:15.445169 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:15.445078    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:59:15.445169 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:15.445124    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:59:15.445295 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:15.445169    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm podName:77cd4cd8-dd31-4511-a6ec-1f162d963772 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:17.445148621 +0000 UTC m=+45.601668890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vl5w" (UID: "77cd4cd8-dd31-4511-a6ec-1f162d963772") : configmap "kube-rbac-proxy" not found
Sep 18 13:59:15.445295 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:15.445201    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls podName:77cd4cd8-dd31-4511-a6ec-1f162d963772 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:17.445190645 +0000 UTC m=+45.601710906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vl5w" (UID: "77cd4cd8-dd31-4511-a6ec-1f162d963772") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:59:15.504272 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:15.504242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb" event={"ID":"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4","Type":"ContainerStarted","Data":"c5ea5d2ec855e82efb315c363b88fd9d1cf8a00d2ae916537ed4966d646c1d93"}
Sep 18 13:59:15.505766 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:15.505746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4" event={"ID":"aad65a15-66da-43c3-98e9-d46a8a109144","Type":"ContainerStarted","Data":"4ef4d291332a5cd9409e5c2d1018061a78be8eba5f3becf06508856ab87fde16"}
Sep 18 13:59:15.507287 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:15.507267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dsbvp" event={"ID":"ab11dd37-11a9-4a40-b00a-267f7080eb18","Type":"ContainerStarted","Data":"6176b52c9733422804b51c296be84b1691dbe9223821bf8a36694dac2f4b88c6"}
Sep 18 13:59:15.525853 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:15.525814    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4" podStartSLOduration=1.943302059 podStartE2EDuration="4.525799034s" totalImagesPullingTime="2.582496975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:15.52530028 +0000 UTC m=+43.681820556" watchObservedRunningTime="2026-09-18 13:59:15.525799034 +0000 UTC m=+43.682319336"
Sep 18 13:59:15.670725 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:15.670630    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-98pdk_336c2a92-68fa-4de6-ba21-fbeab8836c37/node-ca/0.log"
Sep 18 13:59:16.270990 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:16.270963    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bjl8s_d0d7ff07-d797-44c6-8c50-9312493190c1/serve-healthcheck-canary/0.log"
Sep 18 13:59:17.464289 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.464250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:17.464741 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.464350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:17.464741 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:17.464387    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:59:17.464741 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:17.464464    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm podName:77cd4cd8-dd31-4511-a6ec-1f162d963772 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:21.464443115 +0000 UTC m=+49.620963378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vl5w" (UID: "77cd4cd8-dd31-4511-a6ec-1f162d963772") : configmap "kube-rbac-proxy" not found
Sep 18 13:59:17.464741 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:17.464534    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:59:17.464741 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:17.464602    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls podName:77cd4cd8-dd31-4511-a6ec-1f162d963772 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:21.46457829 +0000 UTC m=+49.621098586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vl5w" (UID: "77cd4cd8-dd31-4511-a6ec-1f162d963772") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:59:17.516357 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.516323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb" event={"ID":"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4","Type":"ContainerStarted","Data":"9c269d7178e98c83c933ae7702f6c884c3e69de58c4159939655c39b062a7125"}
Sep 18 13:59:17.534766 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.534717    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb" podStartSLOduration=2.072165103 podStartE2EDuration="4.534698309s" totalImagesPullingTime="2.462533206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:17.534329881 +0000 UTC m=+45.690850188" watchObservedRunningTime="2026-09-18 13:59:17.534698309 +0000 UTC m=+45.691218590"
Sep 18 13:59:17.943230 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.943195    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf"]
Sep 18 13:59:17.972714 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.972682    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf"]
Sep 18 13:59:17.972857 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.972814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf"
Sep 18 13:59:17.975371 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.975346    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:17.975509 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.975370    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 13:59:17.975509 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.975391    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:17.975648 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.975546    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2t2b6\""
Sep 18 13:59:17.975648 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.975572    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 13:59:17.975761 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:17.975741    2563 scope.go:117] "RemoveContainer" containerID="a278ebc8704ca690aa04f954db26042dfad021b33f6f3b951344f8f4c1936777"
Sep 18 13:59:17.975954 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:17.975932    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-v2rtv_openshift-console-operator(d72a0a56-0157-4b87-8ddf-34754f6bdaa3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" podUID="d72a0a56-0157-4b87-8ddf-34754f6bdaa3"
Sep 18 13:59:18.070024 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:18.069980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff46m\" (UniqueName: \"kubernetes.io/projected/3da55f29-e473-4d51-9233-0ee9d2736eed-kube-api-access-ff46m\") pod \"migrator-76794cbd6c-v6drf\" (UID: \"3da55f29-e473-4d51-9233-0ee9d2736eed\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf"
Sep 18 13:59:18.170730 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:18.170699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff46m\" (UniqueName: \"kubernetes.io/projected/3da55f29-e473-4d51-9233-0ee9d2736eed-kube-api-access-ff46m\") pod \"migrator-76794cbd6c-v6drf\" (UID: \"3da55f29-e473-4d51-9233-0ee9d2736eed\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf"
Sep 18 13:59:18.179607 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:18.179580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff46m\" (UniqueName: \"kubernetes.io/projected/3da55f29-e473-4d51-9233-0ee9d2736eed-kube-api-access-ff46m\") pod \"migrator-76794cbd6c-v6drf\" (UID: \"3da55f29-e473-4d51-9233-0ee9d2736eed\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf"
Sep 18 13:59:18.283181 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:18.283084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf"
Sep 18 13:59:18.641641 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:18.641564    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf"]
Sep 18 13:59:18.667677 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:18.667647    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da55f29_e473_4d51_9233_0ee9d2736eed.slice/crio-48d2640744237e89fbac34865549070b9a32a8da9c36cdb45bedfee12827cce8 WatchSource:0}: Error finding container 48d2640744237e89fbac34865549070b9a32a8da9c36cdb45bedfee12827cce8: Status 404 returned error can't find the container with id 48d2640744237e89fbac34865549070b9a32a8da9c36cdb45bedfee12827cce8
Sep 18 13:59:19.523964 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:19.523921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf" event={"ID":"3da55f29-e473-4d51-9233-0ee9d2736eed","Type":"ContainerStarted","Data":"48d2640744237e89fbac34865549070b9a32a8da9c36cdb45bedfee12827cce8"}
Sep 18 13:59:19.525399 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:19.525373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dsbvp" event={"ID":"ab11dd37-11a9-4a40-b00a-267f7080eb18","Type":"ContainerStarted","Data":"99cf511510f5a9553a5cc54e24f0ef87b8744045cf0203aeefa05e2cae5bf040"}
Sep 18 13:59:19.541162 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:19.541099    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-dsbvp" podStartSLOduration=1.816054737 podStartE2EDuration="5.541082013s" totalImagesPullingTime="3.725027276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:19.540926834 +0000 UTC m=+47.697447116" watchObservedRunningTime="2026-09-18 13:59:19.541082013 +0000 UTC m=+47.697602296"
Sep 18 13:59:20.530400 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:20.530347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf" event={"ID":"3da55f29-e473-4d51-9233-0ee9d2736eed","Type":"ContainerStarted","Data":"80043a5089a27267c03a891e1ce4fa3966e3888169b744d3f0b9a91783540122"}
Sep 18 13:59:20.530672 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:20.530397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf" event={"ID":"3da55f29-e473-4d51-9233-0ee9d2736eed","Type":"ContainerStarted","Data":"181f7b901597fa5b8dbdf1bbb4bdb5ffbca6399e186e944e169305ccfff7bd9d"}
Sep 18 13:59:20.547945 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:20.547902    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-v6drf" podStartSLOduration=1.987566396 podStartE2EDuration="3.547887147s" totalImagesPullingTime="1.560320751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:20.54641922 +0000 UTC m=+48.702939499" watchObservedRunningTime="2026-09-18 13:59:20.547887147 +0000 UTC m=+48.704407427"
Sep 18 13:59:21.503174 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.503118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:21.503368 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.503204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:21.503368 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:21.503264    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:59:21.503368 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:21.503309    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:59:21.503368 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:21.503355    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm podName:77cd4cd8-dd31-4511-a6ec-1f162d963772 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:29.503333628 +0000 UTC m=+57.659853939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vl5w" (UID: "77cd4cd8-dd31-4511-a6ec-1f162d963772") : configmap "kube-rbac-proxy" not found
Sep 18 13:59:21.503546 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:21.503378    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls podName:77cd4cd8-dd31-4511-a6ec-1f162d963772 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:29.503368658 +0000 UTC m=+57.659888917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vl5w" (UID: "77cd4cd8-dd31-4511-a6ec-1f162d963772") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:59:21.693809 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.693783    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-bbjhh"]
Sep 18 13:59:21.715687 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.715667    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-bbjhh"]
Sep 18 13:59:21.715789 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.715753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh"
Sep 18 13:59:21.717882 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.717861    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 13:59:21.718834 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.718811    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tx222\""
Sep 18 13:59:21.718951 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.718811    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 13:59:21.718951 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.718869    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 13:59:21.718951 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.718874    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 13:59:21.806515 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.806454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4279f078-4e2a-4abd-a1e9-98c88ecc4501-signing-cabundle\") pod \"service-ca-55f6fc7b86-bbjhh\" (UID: \"4279f078-4e2a-4abd-a1e9-98c88ecc4501\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh"
Sep 18 13:59:21.806600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.806531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlshs\" (UniqueName: \"kubernetes.io/projected/4279f078-4e2a-4abd-a1e9-98c88ecc4501-kube-api-access-nlshs\") pod \"service-ca-55f6fc7b86-bbjhh\" (UID: \"4279f078-4e2a-4abd-a1e9-98c88ecc4501\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh"
Sep 18 13:59:21.806600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.806592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4279f078-4e2a-4abd-a1e9-98c88ecc4501-signing-key\") pod \"service-ca-55f6fc7b86-bbjhh\" (UID: \"4279f078-4e2a-4abd-a1e9-98c88ecc4501\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh"
Sep 18 13:59:21.907093 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.907067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4279f078-4e2a-4abd-a1e9-98c88ecc4501-signing-cabundle\") pod \"service-ca-55f6fc7b86-bbjhh\" (UID: \"4279f078-4e2a-4abd-a1e9-98c88ecc4501\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh"
Sep 18 13:59:21.907187 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.907142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlshs\" (UniqueName: \"kubernetes.io/projected/4279f078-4e2a-4abd-a1e9-98c88ecc4501-kube-api-access-nlshs\") pod \"service-ca-55f6fc7b86-bbjhh\" (UID: \"4279f078-4e2a-4abd-a1e9-98c88ecc4501\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh"
Sep 18 13:59:21.907227 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.907214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4279f078-4e2a-4abd-a1e9-98c88ecc4501-signing-key\") pod \"service-ca-55f6fc7b86-bbjhh\" (UID: \"4279f078-4e2a-4abd-a1e9-98c88ecc4501\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh"
Sep 18 13:59:21.907747 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.907710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4279f078-4e2a-4abd-a1e9-98c88ecc4501-signing-cabundle\") pod \"service-ca-55f6fc7b86-bbjhh\" (UID: \"4279f078-4e2a-4abd-a1e9-98c88ecc4501\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh"
Sep 18 13:59:21.909456 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.909438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4279f078-4e2a-4abd-a1e9-98c88ecc4501-signing-key\") pod \"service-ca-55f6fc7b86-bbjhh\" (UID: \"4279f078-4e2a-4abd-a1e9-98c88ecc4501\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh"
Sep 18 13:59:21.914848 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:21.914827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlshs\" (UniqueName: \"kubernetes.io/projected/4279f078-4e2a-4abd-a1e9-98c88ecc4501-kube-api-access-nlshs\") pod \"service-ca-55f6fc7b86-bbjhh\" (UID: \"4279f078-4e2a-4abd-a1e9-98c88ecc4501\") " pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh"
Sep 18 13:59:22.007800 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:22.007776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:22.007920 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.007905    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:22.007965 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.007959    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls podName:1e8a3b04-7240-4c24-a4c2-133a57d91595 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:38.007944476 +0000 UTC m=+66.164464749 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-fc4t7" (UID: "1e8a3b04-7240-4c24-a4c2-133a57d91595") : secret "cluster-monitoring-operator-tls" not found
Sep 18 13:59:22.024733 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:22.024714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh"
Sep 18 13:59:22.109031 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:22.109005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:22.109155 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:22.109043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:22.109155 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:22.109070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:22.109283 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:22.109168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:22.109283 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.109177    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 13:59:22.109283 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.109189    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:38.10917033 +0000 UTC m=+66.265690605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : configmap references non-existent config key: service-ca.crt
Sep 18 13:59:22.109283 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.109199    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758d69797b-d24nl: secret "image-registry-tls" not found
Sep 18 13:59:22.109283 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.109234    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 13:59:22.109283 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.109251    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:59:22.109283 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.109241    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls podName:ee72d2ae-3ac8-418d-b0a6-3a45e0121d60 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:38.109229238 +0000 UTC m=+66.265749525 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls") pod "image-registry-758d69797b-d24nl" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60") : secret "image-registry-tls" not found
Sep 18 13:59:22.109283 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.109289    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls podName:66650627-a6b5-4d1f-b5c6-3f812de684eb nodeName:}" failed. No retries permitted until 2026-09-18 13:59:38.109281923 +0000 UTC m=+66.265802181 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls") pod "dns-default-k9kwv" (UID: "66650627-a6b5-4d1f-b5c6-3f812de684eb") : secret "dns-default-metrics-tls" not found
Sep 18 13:59:22.109539 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.109301    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs podName:2bc70927-d434-49f3-b09a-dda6af701e30 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:38.109295531 +0000 UTC m=+66.265815789 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs") pod "router-default-764b645cd6-x9x9r" (UID: "2bc70927-d434-49f3-b09a-dda6af701e30") : secret "router-metrics-certs-default" not found
Sep 18 13:59:22.317045 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:22.317022    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-bbjhh"]
Sep 18 13:59:22.320301 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:22.320281    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4279f078_4e2a_4abd_a1e9_98c88ecc4501.slice/crio-12e90815cba4376b7da1a36696ee76582487a462dbd55ae302a7991cb38987e6 WatchSource:0}: Error finding container 12e90815cba4376b7da1a36696ee76582487a462dbd55ae302a7991cb38987e6: Status 404 returned error can't find the container with id 12e90815cba4376b7da1a36696ee76582487a462dbd55ae302a7991cb38987e6
Sep 18 13:59:22.538230 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:22.538199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh" event={"ID":"4279f078-4e2a-4abd-a1e9-98c88ecc4501","Type":"ContainerStarted","Data":"377060fb8e96f568be4cda727fc804829820f570728e722224196c900840577e"}
Sep 18 13:59:22.538353 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:22.538242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh" event={"ID":"4279f078-4e2a-4abd-a1e9-98c88ecc4501","Type":"ContainerStarted","Data":"12e90815cba4376b7da1a36696ee76582487a462dbd55ae302a7991cb38987e6"}
Sep 18 13:59:22.713780 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:22.713756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:22.714097 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.713902    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 13:59:22.714097 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:22.713965    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls podName:5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:38.713946747 +0000 UTC m=+66.870467011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-8hchs" (UID: "5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250") : secret "samples-operator-tls" not found
Sep 18 13:59:29.572088 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:29.572053    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:29.572613 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:29.572113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:29.572613 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:29.572582    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/77cd4cd8-dd31-4511-a6ec-1f162d963772-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:29.574417 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:29.574386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/77cd4cd8-dd31-4511-a6ec-1f162d963772-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vl5w\" (UID: \"77cd4cd8-dd31-4511-a6ec-1f162d963772\") " pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:29.681330 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:29.681303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2vl5w"
Sep 18 13:59:29.963580 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:29.963512    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-bbjhh" podStartSLOduration=8.963492499000001 podStartE2EDuration="8.963492499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:22.556489895 +0000 UTC m=+50.713010177" watchObservedRunningTime="2026-09-18 13:59:29.963492499 +0000 UTC m=+58.120012781"
Sep 18 13:59:29.964196 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:29.964177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2vl5w"]
Sep 18 13:59:29.967642 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:29.967610    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cd4cd8_dd31_4511_a6ec_1f162d963772.slice/crio-93e344fdc5d5e3925c1b3187400f915e4e0aeee44d36c2ff6642d412530cd3f3 WatchSource:0}: Error finding container 93e344fdc5d5e3925c1b3187400f915e4e0aeee44d36c2ff6642d412530cd3f3: Status 404 returned error can't find the container with id 93e344fdc5d5e3925c1b3187400f915e4e0aeee44d36c2ff6642d412530cd3f3
Sep 18 13:59:30.575799 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:30.573363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vl5w" event={"ID":"77cd4cd8-dd31-4511-a6ec-1f162d963772","Type":"ContainerStarted","Data":"8655ab568ac9d03583fcae34268e9139a4c85a90f02f87dbd05c216a2d249679"}
Sep 18 13:59:30.575799 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:30.573477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vl5w" event={"ID":"77cd4cd8-dd31-4511-a6ec-1f162d963772","Type":"ContainerStarted","Data":"93e344fdc5d5e3925c1b3187400f915e4e0aeee44d36c2ff6642d412530cd3f3"}
Sep 18 13:59:31.450503 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:31.450477    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqnf9"
Sep 18 13:59:31.577825 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:31.577794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vl5w" event={"ID":"77cd4cd8-dd31-4511-a6ec-1f162d963772","Type":"ContainerStarted","Data":"8e5aad38fbf227e9cd3579fe882bceeca8aed9b674833f7e9d6da6de782825c6"}
Sep 18 13:59:33.281971 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:33.281941    2563 scope.go:117] "RemoveContainer" containerID="a278ebc8704ca690aa04f954db26042dfad021b33f6f3b951344f8f4c1936777"
Sep 18 13:59:33.585372 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:33.585316    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 13:59:33.585710 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:33.585696    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/1.log"
Sep 18 13:59:33.585761 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:33.585729    2563 generic.go:356] "Generic (PLEG): container finished" podID="d72a0a56-0157-4b87-8ddf-34754f6bdaa3" containerID="0f22c741aabaa6e4c3f3ce68eb38abfb4fadfc65d9c33393012e97412110bed1" exitCode=255
Sep 18 13:59:33.585833 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:33.585805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" event={"ID":"d72a0a56-0157-4b87-8ddf-34754f6bdaa3","Type":"ContainerDied","Data":"0f22c741aabaa6e4c3f3ce68eb38abfb4fadfc65d9c33393012e97412110bed1"}
Sep 18 13:59:33.585891 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:33.585865    2563 scope.go:117] "RemoveContainer" containerID="a278ebc8704ca690aa04f954db26042dfad021b33f6f3b951344f8f4c1936777"
Sep 18 13:59:33.586171 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:33.586153    2563 scope.go:117] "RemoveContainer" containerID="0f22c741aabaa6e4c3f3ce68eb38abfb4fadfc65d9c33393012e97412110bed1"
Sep 18 13:59:33.586359 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:33.586338    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-v2rtv_openshift-console-operator(d72a0a56-0157-4b87-8ddf-34754f6bdaa3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" podUID="d72a0a56-0157-4b87-8ddf-34754f6bdaa3"
Sep 18 13:59:33.587999 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:33.587978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vl5w" event={"ID":"77cd4cd8-dd31-4511-a6ec-1f162d963772","Type":"ContainerStarted","Data":"05a177b321a6fb432df1b9980fbf756a028098b32eb22522edf779c76d2d6e50"}
Sep 18 13:59:33.651047 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:33.651003    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2vl5w" podStartSLOduration=18.171910717 podStartE2EDuration="20.650991109s" totalImagesPullingTime="2.479080392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:59:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:33.650256601 +0000 UTC m=+61.806776881" watchObservedRunningTime="2026-09-18 13:59:33.650991109 +0000 UTC m=+61.807511389"
Sep 18 13:59:34.593290 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:34.593260    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 13:59:37.977173 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:37.977121    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:37.977565 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:37.977529    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 13:59:37.977878 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:37.977846    2563 scope.go:117] "RemoveContainer" containerID="0f22c741aabaa6e4c3f3ce68eb38abfb4fadfc65d9c33393012e97412110bed1"
Sep 18 13:59:37.978047 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:37.978024    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-v2rtv_openshift-console-operator(d72a0a56-0157-4b87-8ddf-34754f6bdaa3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" podUID="d72a0a56-0157-4b87-8ddf-34754f6bdaa3"
Sep 18 13:59:38.041143 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.041101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:59:38.041243 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.041160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:38.041456 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.041421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:59:38.043392 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.043372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e8a3b04-7240-4c24-a4c2-133a57d91595-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-fc4t7\" (UID: \"1e8a3b04-7240-4c24-a4c2-133a57d91595\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:38.045423 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.045405    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:59:38.045893 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.045875    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:59:38.054117 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.054088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bfae7fb-425a-493c-927c-877fd442c666-metrics-certs\") pod \"network-metrics-daemon-bzklt\" (UID: \"8bfae7fb-425a-493c-927c-877fd442c666\") " pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:59:38.054214 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.054151    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6812878a-b037-40b6-89c1-445d49376f65-original-pull-secret\") pod \"global-pull-secret-syncer-pp5sf\" (UID: \"6812878a-b037-40b6-89c1-445d49376f65\") " pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:59:38.092287 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.092266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp5sf"
Sep 18 13:59:38.142560 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.142540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:38.142654 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.142568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:38.142654 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.142585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:38.142654 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.142629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:38.143642 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.143366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bc70927-d434-49f3-b09a-dda6af701e30-service-ca-bundle\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:38.144881 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.144861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66650627-a6b5-4d1f-b5c6-3f812de684eb-metrics-tls\") pod \"dns-default-k9kwv\" (UID: \"66650627-a6b5-4d1f-b5c6-3f812de684eb\") " pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:38.145206 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.145183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls\") pod \"image-registry-758d69797b-d24nl\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") " pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:38.145365 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.145347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bc70927-d434-49f3-b09a-dda6af701e30-metrics-certs\") pod \"router-default-764b645cd6-x9x9r\" (UID: \"2bc70927-d434-49f3-b09a-dda6af701e30\") " pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:38.195294 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.195271    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8cnh9\""
Sep 18 13:59:38.203446 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.203427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bzklt"
Sep 18 13:59:38.243689 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.243667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k48g6\" (UniqueName: \"kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6\") pod \"network-check-target-j5xvn\" (UID: \"9a53946c-99aa-4bb6-83f6-57a08c271b16\") " pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:59:38.245846 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.245826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k48g6\" (UniqueName: \"kubernetes.io/projected/9a53946c-99aa-4bb6-83f6-57a08c271b16-kube-api-access-k48g6\") pod \"network-check-target-j5xvn\" (UID: \"9a53946c-99aa-4bb6-83f6-57a08c271b16\") " pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:59:38.294685 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.294658    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cf2zn\""
Sep 18 13:59:38.300479 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.300458    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"
Sep 18 13:59:38.300744 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.300730    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mtthk\""
Sep 18 13:59:38.309222 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.309204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:38.309446 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.309431    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w5qd9\""
Sep 18 13:59:38.315161 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.315146    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fh88p\""
Sep 18 13:59:38.317896 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.317883    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:38.323507 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.323494    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:38.397053 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.397023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pp5sf"]
Sep 18 13:59:38.404475 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:38.404401    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6812878a_b037_40b6_89c1_445d49376f65.slice/crio-233fdfccae9193365c35a605980cbfc06c5292b80161c02b6b642fd3c34e345a WatchSource:0}: Error finding container 233fdfccae9193365c35a605980cbfc06c5292b80161c02b6b642fd3c34e345a: Status 404 returned error can't find the container with id 233fdfccae9193365c35a605980cbfc06c5292b80161c02b6b642fd3c34e345a
Sep 18 13:59:38.497947 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.497898    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bzklt"]
Sep 18 13:59:38.500555 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:38.500530    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bfae7fb_425a_493c_927c_877fd442c666.slice/crio-79d924ef3c4b6280145a8fd477b12e74a915972c0dd8f7beb26bcaba5e3900a6 WatchSource:0}: Error finding container 79d924ef3c4b6280145a8fd477b12e74a915972c0dd8f7beb26bcaba5e3900a6: Status 404 returned error can't find the container with id 79d924ef3c4b6280145a8fd477b12e74a915972c0dd8f7beb26bcaba5e3900a6
Sep 18 13:59:38.500665 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.500608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-z6j5p\""
Sep 18 13:59:38.508988 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.508973    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:59:38.591962 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.591937    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7"]
Sep 18 13:59:38.595376 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:38.595344    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8a3b04_7240_4c24_a4c2_133a57d91595.slice/crio-e5c5c778991619af71402d660911e0c02871d687dcaaeb8190f3ef73c97aa4e4 WatchSource:0}: Error finding container e5c5c778991619af71402d660911e0c02871d687dcaaeb8190f3ef73c97aa4e4: Status 404 returned error can't find the container with id e5c5c778991619af71402d660911e0c02871d687dcaaeb8190f3ef73c97aa4e4
Sep 18 13:59:38.609456 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.609432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7" event={"ID":"1e8a3b04-7240-4c24-a4c2-133a57d91595","Type":"ContainerStarted","Data":"e5c5c778991619af71402d660911e0c02871d687dcaaeb8190f3ef73c97aa4e4"}
Sep 18 13:59:38.610540 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.610518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzklt" event={"ID":"8bfae7fb-425a-493c-927c-877fd442c666","Type":"ContainerStarted","Data":"79d924ef3c4b6280145a8fd477b12e74a915972c0dd8f7beb26bcaba5e3900a6"}
Sep 18 13:59:38.611582 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.611555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pp5sf" event={"ID":"6812878a-b037-40b6-89c1-445d49376f65","Type":"ContainerStarted","Data":"233fdfccae9193365c35a605980cbfc06c5292b80161c02b6b642fd3c34e345a"}
Sep 18 13:59:38.617440 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.617422    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-764b645cd6-x9x9r"]
Sep 18 13:59:38.619366 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:38.619345    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc70927_d434_49f3_b09a_dda6af701e30.slice/crio-8565beed629861e05513b01216db20a3b558cba1bb1e8e687240c9ebbaaf97e1 WatchSource:0}: Error finding container 8565beed629861e05513b01216db20a3b558cba1bb1e8e687240c9ebbaaf97e1: Status 404 returned error can't find the container with id 8565beed629861e05513b01216db20a3b558cba1bb1e8e687240c9ebbaaf97e1
Sep 18 13:59:38.648954 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.648932    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k9kwv"]
Sep 18 13:59:38.652198 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:38.652177    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66650627_a6b5_4d1f_b5c6_3f812de684eb.slice/crio-1792fccf41819e9c16cad0fcac86db6a28738d1d4aed4a7bff9db029a5830dad WatchSource:0}: Error finding container 1792fccf41819e9c16cad0fcac86db6a28738d1d4aed4a7bff9db029a5830dad: Status 404 returned error can't find the container with id 1792fccf41819e9c16cad0fcac86db6a28738d1d4aed4a7bff9db029a5830dad
Sep 18 13:59:38.671933 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.671914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-758d69797b-d24nl"]
Sep 18 13:59:38.674013 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:38.673993    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee72d2ae_3ac8_418d_b0a6_3a45e0121d60.slice/crio-c4efa3b9ef3b5b03b3aecaa6dfb2be64b15b3ffd4efc923e190980bd5a19f56b WatchSource:0}: Error finding container c4efa3b9ef3b5b03b3aecaa6dfb2be64b15b3ffd4efc923e190980bd5a19f56b: Status 404 returned error can't find the container with id c4efa3b9ef3b5b03b3aecaa6dfb2be64b15b3ffd4efc923e190980bd5a19f56b
Sep 18 13:59:38.747363 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.747339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:38.749964 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.749914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-8hchs\" (UID: \"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:38.804497 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.804476    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5xvn"]
Sep 18 13:59:38.807273 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:38.807252    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a53946c_99aa_4bb6_83f6_57a08c271b16.slice/crio-3cb23e00e7c4cc59d4ffb3de4d75198d94434c0e4ebc5b2721ac721911e85254 WatchSource:0}: Error finding container 3cb23e00e7c4cc59d4ffb3de4d75198d94434c0e4ebc5b2721ac721911e85254: Status 404 returned error can't find the container with id 3cb23e00e7c4cc59d4ffb3de4d75198d94434c0e4ebc5b2721ac721911e85254
Sep 18 13:59:38.893002 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.892978    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4ks8d\""
Sep 18 13:59:38.901440 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:38.901414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"
Sep 18 13:59:39.183823 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.183742    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs"]
Sep 18 13:59:39.618872 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.618722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9kwv" event={"ID":"66650627-a6b5-4d1f-b5c6-3f812de684eb","Type":"ContainerStarted","Data":"1792fccf41819e9c16cad0fcac86db6a28738d1d4aed4a7bff9db029a5830dad"}
Sep 18 13:59:39.620471 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.620408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs" event={"ID":"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250","Type":"ContainerStarted","Data":"a191832dd841463b31eeff2e0b06b76a28cfe378a2d9965d7d9c43f3ca6aa98c"}
Sep 18 13:59:39.622804 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.622773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-764b645cd6-x9x9r" event={"ID":"2bc70927-d434-49f3-b09a-dda6af701e30","Type":"ContainerStarted","Data":"33df785e17474fc135b72aa140cd2b220f18b43dd18214a60f526c0da56ebe3d"}
Sep 18 13:59:39.622936 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.622827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-764b645cd6-x9x9r" event={"ID":"2bc70927-d434-49f3-b09a-dda6af701e30","Type":"ContainerStarted","Data":"8565beed629861e05513b01216db20a3b558cba1bb1e8e687240c9ebbaaf97e1"}
Sep 18 13:59:39.626146 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.626038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5xvn" event={"ID":"9a53946c-99aa-4bb6-83f6-57a08c271b16","Type":"ContainerStarted","Data":"89cd4b6fc2807dd1750f6323952683882fd1c601e554ae4005b6fc2d53138ecb"}
Sep 18 13:59:39.626146 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.626081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5xvn" event={"ID":"9a53946c-99aa-4bb6-83f6-57a08c271b16","Type":"ContainerStarted","Data":"3cb23e00e7c4cc59d4ffb3de4d75198d94434c0e4ebc5b2721ac721911e85254"}
Sep 18 13:59:39.626790 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.626576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 13:59:39.629103 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.629078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758d69797b-d24nl" event={"ID":"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60","Type":"ContainerStarted","Data":"65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd"}
Sep 18 13:59:39.629230 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.629121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758d69797b-d24nl" event={"ID":"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60","Type":"ContainerStarted","Data":"c4efa3b9ef3b5b03b3aecaa6dfb2be64b15b3ffd4efc923e190980bd5a19f56b"}
Sep 18 13:59:39.629627 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.629608    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 13:59:39.647150 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.646674    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-764b645cd6-x9x9r" podStartSLOduration=34.646656054 podStartE2EDuration="34.646656054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:39.644976568 +0000 UTC m=+67.801496850" watchObservedRunningTime="2026-09-18 13:59:39.646656054 +0000 UTC m=+67.803176329"
Sep 18 13:59:39.667318 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.666492    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j5xvn" podStartSLOduration=67.666475247 podStartE2EDuration="1m7.666475247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:39.665458968 +0000 UTC m=+67.821979249" watchObservedRunningTime="2026-09-18 13:59:39.666475247 +0000 UTC m=+67.822995529"
Sep 18 13:59:39.688320 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:39.687891    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-758d69797b-d24nl" podStartSLOduration=67.687872638 podStartE2EDuration="1m7.687872638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:39.685950986 +0000 UTC m=+67.842471268" watchObservedRunningTime="2026-09-18 13:59:39.687872638 +0000 UTC m=+67.844392921"
Sep 18 13:59:40.310297 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:40.310224    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:40.313002 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:40.312946    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:40.636210 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:40.636047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzklt" event={"ID":"8bfae7fb-425a-493c-927c-877fd442c666","Type":"ContainerStarted","Data":"62f263a0b5c26f9ba72a68ecf31d8260bd85e9904bab6d1f0a3fc21cc19ebbea"}
Sep 18 13:59:40.636210 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:40.636114    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bzklt" event={"ID":"8bfae7fb-425a-493c-927c-877fd442c666","Type":"ContainerStarted","Data":"518e097dbe5e47ec1e770eb0085b4767cd317b0b04ca07b19b08b2c12ae6bb9e"}
Sep 18 13:59:40.636542 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:40.636518    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:40.637528 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:40.637504    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-764b645cd6-x9x9r"
Sep 18 13:59:40.652266 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:40.652223    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bzklt" podStartSLOduration=67.329930476 podStartE2EDuration="1m8.652209705s" totalImagesPullingTime="1.322279229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:40.65040913 +0000 UTC m=+68.806929410" watchObservedRunningTime="2026-09-18 13:59:40.652209705 +0000 UTC m=+68.808729988"
Sep 18 13:59:43.651694 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.651653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9kwv" event={"ID":"66650627-a6b5-4d1f-b5c6-3f812de684eb","Type":"ContainerStarted","Data":"48a261174fed9cd52e03b59be15628dfb2aa87088ec207254fb060ceaaf86fb8"}
Sep 18 13:59:43.652192 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.651747    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k9kwv" event={"ID":"66650627-a6b5-4d1f-b5c6-3f812de684eb","Type":"ContainerStarted","Data":"d0e8337b951544283752611b5f86ede7e5b9361c0ca0ac03de3f49d35434fda4"}
Sep 18 13:59:43.652192 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.651794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:43.653584 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.653555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs" event={"ID":"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250","Type":"ContainerStarted","Data":"8e4faea39b0d6ad3e7879a3f2a0466913c1b1e17b634df964307b0fe5f941d75"}
Sep 18 13:59:43.653719 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.653598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs" event={"ID":"5ad3d178-9fcb-4e99-bb2f-80d7dfa6e250","Type":"ContainerStarted","Data":"f4ee0b27bbc5b13137b1db32cf72a9c0516feeae639f27bcf7e4d0f591614c4a"}
Sep 18 13:59:43.655040 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.655015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7" event={"ID":"1e8a3b04-7240-4c24-a4c2-133a57d91595","Type":"ContainerStarted","Data":"ca0c6722ffbb572150350716db656235b28fd05f637ad30bb1250deeae2bd3cc"}
Sep 18 13:59:43.656477 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.656442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pp5sf" event={"ID":"6812878a-b037-40b6-89c1-445d49376f65","Type":"ContainerStarted","Data":"48bb5e31a48fa06ba7d3f7441c6ef955c72490e071f88024adec5ad598f0b7e4"}
Sep 18 13:59:43.765547 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.765499    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k9kwv" podStartSLOduration=33.373410889 podStartE2EDuration="37.7654833s" totalImagesPullingTime="4.392072411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:43.727111192 +0000 UTC m=+71.883631474" watchObservedRunningTime="2026-09-18 13:59:43.7654833 +0000 UTC m=+71.922003581"
Sep 18 13:59:43.797466 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.797422    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-fc4t7" podStartSLOduration=34.353949264 podStartE2EDuration="38.797407938s" totalImagesPullingTime="4.443458674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:43.795333427 +0000 UTC m=+71.951853709" watchObservedRunningTime="2026-09-18 13:59:43.797407938 +0000 UTC m=+71.953928218"
Sep 18 13:59:43.797623 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.797577    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pp5sf" podStartSLOduration=67.105359431 podStartE2EDuration="1m11.797571349s" totalImagesPullingTime="4.692211918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:43.765160252 +0000 UTC m=+71.921680534" watchObservedRunningTime="2026-09-18 13:59:43.797571349 +0000 UTC m=+71.954091629"
Sep 18 13:59:43.835654 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.835624    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x"]
Sep 18 13:59:43.835935 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.835900    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-8hchs" podStartSLOduration=34.040957901 podStartE2EDuration="37.835891784s" totalImagesPullingTime="3.794933883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:43.83509976 +0000 UTC m=+71.991620040" watchObservedRunningTime="2026-09-18 13:59:43.835891784 +0000 UTC m=+71.992412090"
Sep 18 13:59:43.840042 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.840027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x"
Sep 18 13:59:43.843035 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.843015    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-k4rsm\""
Sep 18 13:59:43.843550 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.843534    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 13:59:43.843777 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.843764    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 13:59:43.857168 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.857145    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x"]
Sep 18 13:59:43.955178 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.955120    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42"]
Sep 18 13:59:43.958245 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.958223    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42"
Sep 18 13:59:43.962073 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.962055    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hc6fc\""
Sep 18 13:59:43.962294 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.962279    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 13:59:43.976324 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.976301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42"]
Sep 18 13:59:43.992248 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.992220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff7fb1d3-fbc3-4397-afe8-fa68ae822717-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-lpn5x\" (UID: \"ff7fb1d3-fbc3-4397-afe8-fa68ae822717\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x"
Sep 18 13:59:43.992362 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:43.992276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ff7fb1d3-fbc3-4397-afe8-fa68ae822717-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-lpn5x\" (UID: \"ff7fb1d3-fbc3-4397-afe8-fa68ae822717\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x"
Sep 18 13:59:44.092750 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.092702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff7fb1d3-fbc3-4397-afe8-fa68ae822717-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-lpn5x\" (UID: \"ff7fb1d3-fbc3-4397-afe8-fa68ae822717\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x"
Sep 18 13:59:44.092750 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.092756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6782b4e4-dfd5-43b9-ab12-d3c049f0c930-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-r5l42\" (UID: \"6782b4e4-dfd5-43b9-ab12-d3c049f0c930\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42"
Sep 18 13:59:44.092969 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.092794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ff7fb1d3-fbc3-4397-afe8-fa68ae822717-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-lpn5x\" (UID: \"ff7fb1d3-fbc3-4397-afe8-fa68ae822717\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x"
Sep 18 13:59:44.093488 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.093463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ff7fb1d3-fbc3-4397-afe8-fa68ae822717-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-lpn5x\" (UID: \"ff7fb1d3-fbc3-4397-afe8-fa68ae822717\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x"
Sep 18 13:59:44.095141 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.095106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ff7fb1d3-fbc3-4397-afe8-fa68ae822717-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-lpn5x\" (UID: \"ff7fb1d3-fbc3-4397-afe8-fa68ae822717\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x"
Sep 18 13:59:44.149087 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.149062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x"
Sep 18 13:59:44.193601 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.193570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6782b4e4-dfd5-43b9-ab12-d3c049f0c930-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-r5l42\" (UID: \"6782b4e4-dfd5-43b9-ab12-d3c049f0c930\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42"
Sep 18 13:59:44.195865 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.195842    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6782b4e4-dfd5-43b9-ab12-d3c049f0c930-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-r5l42\" (UID: \"6782b4e4-dfd5-43b9-ab12-d3c049f0c930\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42"
Sep 18 13:59:44.266594 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.266563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42"
Sep 18 13:59:44.458481 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.458458    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x"]
Sep 18 13:59:44.460848 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:44.460822    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7fb1d3_fbc3_4397_afe8_fa68ae822717.slice/crio-66ba3d7a929f1aec43fb08649f4c2f7da5303dcf54d8135332a282cb65577898 WatchSource:0}: Error finding container 66ba3d7a929f1aec43fb08649f4c2f7da5303dcf54d8135332a282cb65577898: Status 404 returned error can't find the container with id 66ba3d7a929f1aec43fb08649f4c2f7da5303dcf54d8135332a282cb65577898
Sep 18 13:59:44.554633 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.554609    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42"]
Sep 18 13:59:44.556810 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:44.556788    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6782b4e4_dfd5_43b9_ab12_d3c049f0c930.slice/crio-eb6fa27f907c8877a27315b964327a8e52449843c44dc20a0162ed1849bf36fc WatchSource:0}: Error finding container eb6fa27f907c8877a27315b964327a8e52449843c44dc20a0162ed1849bf36fc: Status 404 returned error can't find the container with id eb6fa27f907c8877a27315b964327a8e52449843c44dc20a0162ed1849bf36fc
Sep 18 13:59:44.661389 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.661353    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x" event={"ID":"ff7fb1d3-fbc3-4397-afe8-fa68ae822717","Type":"ContainerStarted","Data":"66ba3d7a929f1aec43fb08649f4c2f7da5303dcf54d8135332a282cb65577898"}
Sep 18 13:59:44.662525 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:44.662498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42" event={"ID":"6782b4e4-dfd5-43b9-ab12-d3c049f0c930","Type":"ContainerStarted","Data":"eb6fa27f907c8877a27315b964327a8e52449843c44dc20a0162ed1849bf36fc"}
Sep 18 13:59:46.671330 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:46.671295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42" event={"ID":"6782b4e4-dfd5-43b9-ab12-d3c049f0c930","Type":"ContainerStarted","Data":"6b24fa55c4ff4c3d524749572f7f6cdde95c8c77550f0106e3f6a0e23975d812"}
Sep 18 13:59:46.671759 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:46.671400    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42"
Sep 18 13:59:46.673040 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:46.673008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x" event={"ID":"ff7fb1d3-fbc3-4397-afe8-fa68ae822717","Type":"ContainerStarted","Data":"ff565204e77e8cca9c472c57420f4a428393a27f63b52b4cc8fbaac956ffb2c7"}
Sep 18 13:59:46.676265 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:46.676236    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42"
Sep 18 13:59:46.689040 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:46.688995    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-r5l42" podStartSLOduration=1.843369777 podStartE2EDuration="3.688980969s" totalImagesPullingTime="1.845611192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:46.688517331 +0000 UTC m=+74.845037612" watchObservedRunningTime="2026-09-18 13:59:46.688980969 +0000 UTC m=+74.845501250"
Sep 18 13:59:46.729890 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:46.729839    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-lpn5x" podStartSLOduration=1.790421939 podStartE2EDuration="3.729825573s" totalImagesPullingTime="1.939403634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:46.728671099 +0000 UTC m=+74.885191413" watchObservedRunningTime="2026-09-18 13:59:46.729825573 +0000 UTC m=+74.886345903"
Sep 18 13:59:50.282533 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:50.282495    2563 scope.go:117] "RemoveContainer" containerID="0f22c741aabaa6e4c3f3ce68eb38abfb4fadfc65d9c33393012e97412110bed1"
Sep 18 13:59:50.283111 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:50.282669    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-v2rtv_openshift-console-operator(d72a0a56-0157-4b87-8ddf-34754f6bdaa3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" podUID="d72a0a56-0157-4b87-8ddf-34754f6bdaa3"
Sep 18 13:59:52.350274 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.350237    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dt75n"]
Sep 18 13:59:52.353879 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.353860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.360872 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.360825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 13:59:52.361077 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.361018    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 13:59:52.361315 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.361297    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fvvcw\""
Sep 18 13:59:52.361761 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.361739    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 13:59:52.362701 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.362684    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 13:59:52.457313 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.457285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-metrics-client-ca\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.457460 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.457325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-root\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.457460 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.457351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmghs\" (UniqueName: \"kubernetes.io/projected/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-kube-api-access-hmghs\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.457460 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.457373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-sys\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.457565 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.457461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-tls\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.457565 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.457518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.457630 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.457568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-accelerators-collector-config\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.457663 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.457621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-textfile\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.457738 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.457716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-wtmp\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559010 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.558974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-wtmp\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559192 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-metrics-client-ca\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559192 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-root\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559192 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-root\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559192 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmghs\" (UniqueName: \"kubernetes.io/projected/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-kube-api-access-hmghs\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559422 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-wtmp\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559422 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-sys\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559422 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-tls\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559422 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559422 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-sys\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559422 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-accelerators-collector-config\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559738 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-textfile\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559738 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-metrics-client-ca\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559813 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-textfile\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.559952 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.559933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-accelerators-collector-config\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.560015 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:52.559999    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 13:59:52.560089 ip-10-0-139-55 kubenswrapper[2563]: E0918 13:59:52.560079    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-tls podName:08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9 nodeName:}" failed. No retries permitted until 2026-09-18 13:59:53.060059293 +0000 UTC m=+81.216579558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-tls") pod "node-exporter-dt75n" (UID: "08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9") : secret "node-exporter-tls" not found
Sep 18 13:59:52.561909 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.561880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:52.569272 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:52.569250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmghs\" (UniqueName: \"kubernetes.io/projected/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-kube-api-access-hmghs\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:53.064212 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:53.064174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-tls\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:53.066503 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:53.066481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9-node-exporter-tls\") pod \"node-exporter-dt75n\" (UID: \"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9\") " pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:53.263588 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:53.263556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dt75n"
Sep 18 13:59:53.274900 ip-10-0-139-55 kubenswrapper[2563]: W0918 13:59:53.274853    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c9cfc7_d31b_4ebd_9a96_3ec8f3d54ec9.slice/crio-f3c6f2f6438af3be2e9d92a8120d2ba30a4b8d3a675e25b2ac59b0b77a707921 WatchSource:0}: Error finding container f3c6f2f6438af3be2e9d92a8120d2ba30a4b8d3a675e25b2ac59b0b77a707921: Status 404 returned error can't find the container with id f3c6f2f6438af3be2e9d92a8120d2ba30a4b8d3a675e25b2ac59b0b77a707921
Sep 18 13:59:53.665768 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:53.665734    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k9kwv"
Sep 18 13:59:53.701357 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:53.701292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dt75n" event={"ID":"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9","Type":"ContainerStarted","Data":"f3c6f2f6438af3be2e9d92a8120d2ba30a4b8d3a675e25b2ac59b0b77a707921"}
Sep 18 13:59:54.706593 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:54.706560    2563 generic.go:356] "Generic (PLEG): container finished" podID="08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9" containerID="49e361c2be63d87fd0569a96cb1d86aa07b4157cb44874a5871c56e293f4cca0" exitCode=0
Sep 18 13:59:54.707045 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:54.706646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dt75n" event={"ID":"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9","Type":"ContainerDied","Data":"49e361c2be63d87fd0569a96cb1d86aa07b4157cb44874a5871c56e293f4cca0"}
Sep 18 13:59:55.712918 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:55.712876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dt75n" event={"ID":"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9","Type":"ContainerStarted","Data":"6c626010546586018cfd82f60a2c9922ca52efef1e7d009a1b6230175027f343"}
Sep 18 13:59:55.713329 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:55.712976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dt75n" event={"ID":"08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9","Type":"ContainerStarted","Data":"40359daf83f9c824d86c48f49d2792e1a8fb74f45a3a9cc1ba9596b266abb38b"}
Sep 18 13:59:55.735912 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:55.735862    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dt75n" podStartSLOduration=2.8350989 podStartE2EDuration="3.735846608s" totalImagesPullingTime="841.225271ms" totalInitContainerRuntime="59.522437ms" isStatefulPod=true podCreationTimestamp="2026-09-18 13:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:55.735447192 +0000 UTC m=+83.891967473" watchObservedRunningTime="2026-09-18 13:59:55.735846608 +0000 UTC m=+83.892366889"
Sep 18 13:59:58.322208 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:58.322168    2563 patch_prober.go:28] interesting pod/image-registry-758d69797b-d24nl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 13:59:58.322600 ip-10-0-139-55 kubenswrapper[2563]: I0918 13:59:58.322226    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-758d69797b-d24nl" podUID="ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:00:01.282486 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:01.282451    2563 scope.go:117] "RemoveContainer" containerID="0f22c741aabaa6e4c3f3ce68eb38abfb4fadfc65d9c33393012e97412110bed1"
Sep 18 14:00:01.646220 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:01.646118    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 14:00:01.737471 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:01.737446    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:00:01.737606 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:01.737537    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" event={"ID":"d72a0a56-0157-4b87-8ddf-34754f6bdaa3","Type":"ContainerStarted","Data":"3e9b7884b57c04db15bcab88c4ce3d8c5e1090460ff2d44a1388fe0c83a54f4f"}
Sep 18 14:00:01.737821 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:01.737801    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 14:00:01.755878 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:01.755828    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv" podStartSLOduration=50.894369255 podStartE2EDuration="54.755813416s" totalImagesPullingTime="3.861444161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:01.75537278 +0000 UTC m=+89.911893061" watchObservedRunningTime="2026-09-18 14:00:01.755813416 +0000 UTC m=+89.912333738"
Sep 18 14:00:02.448965 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:02.448934    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-v2rtv"
Sep 18 14:00:02.628007 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:02.627976    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-nn5ds"]
Sep 18 14:00:02.631916 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:02.631897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-nn5ds"
Sep 18 14:00:02.634335 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:02.634305    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 14:00:02.634434 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:02.634408    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zsst8\""
Sep 18 14:00:02.634434 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:02.634421    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 14:00:02.645152 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:02.645116    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-nn5ds"]
Sep 18 14:00:02.740769 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:02.740684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srf9v\" (UniqueName: \"kubernetes.io/projected/23f2129e-d074-425b-aa31-7b53bd82749d-kube-api-access-srf9v\") pod \"downloads-6594f4c75-nn5ds\" (UID: \"23f2129e-d074-425b-aa31-7b53bd82749d\") " pod="openshift-console/downloads-6594f4c75-nn5ds"
Sep 18 14:00:02.841954 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:02.841912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srf9v\" (UniqueName: \"kubernetes.io/projected/23f2129e-d074-425b-aa31-7b53bd82749d-kube-api-access-srf9v\") pod \"downloads-6594f4c75-nn5ds\" (UID: \"23f2129e-d074-425b-aa31-7b53bd82749d\") " pod="openshift-console/downloads-6594f4c75-nn5ds"
Sep 18 14:00:02.851243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:02.851076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srf9v\" (UniqueName: \"kubernetes.io/projected/23f2129e-d074-425b-aa31-7b53bd82749d-kube-api-access-srf9v\") pod \"downloads-6594f4c75-nn5ds\" (UID: \"23f2129e-d074-425b-aa31-7b53bd82749d\") " pod="openshift-console/downloads-6594f4c75-nn5ds"
Sep 18 14:00:02.940970 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:02.940939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-nn5ds"
Sep 18 14:00:03.221679 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:03.221646    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-nn5ds"]
Sep 18 14:00:03.224479 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:00:03.224453    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f2129e_d074_425b_aa31_7b53bd82749d.slice/crio-ef29459df3e22377164891c48cb8f47cc2fd66bf53e1441a46f1c01f5289b9fd WatchSource:0}: Error finding container ef29459df3e22377164891c48cb8f47cc2fd66bf53e1441a46f1c01f5289b9fd: Status 404 returned error can't find the container with id ef29459df3e22377164891c48cb8f47cc2fd66bf53e1441a46f1c01f5289b9fd
Sep 18 14:00:03.746644 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:03.746611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-nn5ds" event={"ID":"23f2129e-d074-425b-aa31-7b53bd82749d","Type":"ContainerStarted","Data":"ef29459df3e22377164891c48cb8f47cc2fd66bf53e1441a46f1c01f5289b9fd"}
Sep 18 14:00:07.352173 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:07.352122    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-758d69797b-d24nl"]
Sep 18 14:00:11.645025 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.644991    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j5xvn"
Sep 18 14:00:11.701508 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.701473    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54955bb4c4-69dzf"]
Sep 18 14:00:11.707426 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.707402    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.710812 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.710588    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 14:00:11.710812 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.710603    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 14:00:11.711001 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.710865    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 14:00:11.711001 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.710897    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 14:00:11.711213 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.711156    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 14:00:11.711288 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.711159    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pbm9v\""
Sep 18 14:00:11.715446 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.715421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54955bb4c4-69dzf"]
Sep 18 14:00:11.813842 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.813804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-oauth-config\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.814018 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.813849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-oauth-serving-cert\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.814018 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.813873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-service-ca\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.814018 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.813951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-serving-cert\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.814018 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.813996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-console-config\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.814018 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.814017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sb59\" (UniqueName: \"kubernetes.io/projected/b770194f-27a5-4721-a18d-f0ff4633f808-kube-api-access-7sb59\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.914553 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.914472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-console-config\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.914553 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.914522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sb59\" (UniqueName: \"kubernetes.io/projected/b770194f-27a5-4721-a18d-f0ff4633f808-kube-api-access-7sb59\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.914776 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.914581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-oauth-config\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.914776 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.914617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-oauth-serving-cert\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.914776 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.914639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-service-ca\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.914776 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.914670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-serving-cert\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.915355 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.915325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-oauth-serving-cert\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.915474 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.915325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-console-config\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.915553 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.915533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-service-ca\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.917815 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.917773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-oauth-config\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.917919 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.917879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-serving-cert\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:11.924027 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:11.924003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sb59\" (UniqueName: \"kubernetes.io/projected/b770194f-27a5-4721-a18d-f0ff4633f808-kube-api-access-7sb59\") pod \"console-54955bb4c4-69dzf\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") " pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:12.019970 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:12.019935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:12.313119 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:12.313090    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54955bb4c4-69dzf"]
Sep 18 14:00:12.316001 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:00:12.315975    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb770194f_27a5_4721_a18d_f0ff4633f808.slice/crio-b817bc1c9e4469a2e7eafda0549d9c6333db8224bed2cd0c6d14418773ad61f9 WatchSource:0}: Error finding container b817bc1c9e4469a2e7eafda0549d9c6333db8224bed2cd0c6d14418773ad61f9: Status 404 returned error can't find the container with id b817bc1c9e4469a2e7eafda0549d9c6333db8224bed2cd0c6d14418773ad61f9
Sep 18 14:00:12.790381 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:12.790344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54955bb4c4-69dzf" event={"ID":"b770194f-27a5-4721-a18d-f0ff4633f808","Type":"ContainerStarted","Data":"b817bc1c9e4469a2e7eafda0549d9c6333db8224bed2cd0c6d14418773ad61f9"}
Sep 18 14:00:19.135219 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.135184    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-94b9b8989-n9tdx"]
Sep 18 14:00:19.137510 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.137489    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.145627 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.145600    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 14:00:19.151411 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.151359    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-94b9b8989-n9tdx"]
Sep 18 14:00:19.183688 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.183652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-oauth-config\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.183816 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.183695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-console-config\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.183816 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.183766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-service-ca\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.183816 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.183784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-oauth-serving-cert\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.183816 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.183810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-trusted-ca-bundle\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.183980 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.183884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-serving-cert\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.183980 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.183923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv6jj\" (UniqueName: \"kubernetes.io/projected/8ef57b54-12fe-4b31-a658-d84fca154106-kube-api-access-pv6jj\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.284601 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.284563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-oauth-config\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.284601 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.284604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-console-config\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.284840 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.284663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-service-ca\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.284840 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.284686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-oauth-serving-cert\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.284840 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.284712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-trusted-ca-bundle\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.284978 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.284848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-serving-cert\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.284978 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.284899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv6jj\" (UniqueName: \"kubernetes.io/projected/8ef57b54-12fe-4b31-a658-d84fca154106-kube-api-access-pv6jj\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.285559 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.285509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-oauth-serving-cert\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.285688 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.285585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-service-ca\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.285688 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.285651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-console-config\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.285946 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.285920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-trusted-ca-bundle\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.287478 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.287454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-oauth-config\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.287848 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.287828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-serving-cert\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.293265 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.293245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv6jj\" (UniqueName: \"kubernetes.io/projected/8ef57b54-12fe-4b31-a658-d84fca154106-kube-api-access-pv6jj\") pod \"console-94b9b8989-n9tdx\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") " pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:19.447517 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:19.447486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:20.827555 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:20.827468    2563 generic.go:356] "Generic (PLEG): container finished" podID="be1f9fd0-371e-4d59-a868-42ba681905e7" containerID="bc8a124a1c411aabd24f955534faf9ad6a9c67db7af62c8f5830a793f1e1ee1c" exitCode=0
Sep 18 14:00:20.827555 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:20.827524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tj226" event={"ID":"be1f9fd0-371e-4d59-a868-42ba681905e7","Type":"ContainerDied","Data":"bc8a124a1c411aabd24f955534faf9ad6a9c67db7af62c8f5830a793f1e1ee1c"}
Sep 18 14:00:20.828022 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:20.828004    2563 scope.go:117] "RemoveContainer" containerID="bc8a124a1c411aabd24f955534faf9ad6a9c67db7af62c8f5830a793f1e1ee1c"
Sep 18 14:00:23.263052 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:23.263022    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-94b9b8989-n9tdx"]
Sep 18 14:00:23.265544 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:00:23.265517    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef57b54_12fe_4b31_a658_d84fca154106.slice/crio-de8289ee8c0876e157315203a71d80fd4abf7e02066e62ee44f9f43acf132a49 WatchSource:0}: Error finding container de8289ee8c0876e157315203a71d80fd4abf7e02066e62ee44f9f43acf132a49: Status 404 returned error can't find the container with id de8289ee8c0876e157315203a71d80fd4abf7e02066e62ee44f9f43acf132a49
Sep 18 14:00:23.842959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:23.842910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-nn5ds" event={"ID":"23f2129e-d074-425b-aa31-7b53bd82749d","Type":"ContainerStarted","Data":"69c88094b4c21c1c9ad2c112cd5c528ad819252a9b931c42947f5484b7977a09"}
Sep 18 14:00:23.844892 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:23.844861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54955bb4c4-69dzf" event={"ID":"b770194f-27a5-4721-a18d-f0ff4633f808","Type":"ContainerStarted","Data":"8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef"}
Sep 18 14:00:23.847197 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:23.847166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tj226" event={"ID":"be1f9fd0-371e-4d59-a868-42ba681905e7","Type":"ContainerStarted","Data":"cc6c2929bcb46b441343b1efe6be29a6e7fad9b34039634196e92fe27e0dd8db"}
Sep 18 14:00:23.849053 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:23.849023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94b9b8989-n9tdx" event={"ID":"8ef57b54-12fe-4b31-a658-d84fca154106","Type":"ContainerStarted","Data":"6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99"}
Sep 18 14:00:23.849176 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:23.849072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94b9b8989-n9tdx" event={"ID":"8ef57b54-12fe-4b31-a658-d84fca154106","Type":"ContainerStarted","Data":"de8289ee8c0876e157315203a71d80fd4abf7e02066e62ee44f9f43acf132a49"}
Sep 18 14:00:23.864435 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:23.864377    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-nn5ds" podStartSLOduration=2.092537317 podStartE2EDuration="21.864358293s" totalImagesPullingTime="19.771820976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:23.862030988 +0000 UTC m=+112.018551269" watchObservedRunningTime="2026-09-18 14:00:23.864358293 +0000 UTC m=+112.020878574"
Sep 18 14:00:23.888153 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:23.888059    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54955bb4c4-69dzf" podStartSLOduration=2.140856478 podStartE2EDuration="12.888036797s" totalImagesPullingTime="10.747180319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:23.88658221 +0000 UTC m=+112.043102493" watchObservedRunningTime="2026-09-18 14:00:23.888036797 +0000 UTC m=+112.044557078"
Sep 18 14:00:23.914619 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:23.914551    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-94b9b8989-n9tdx" podStartSLOduration=4.914515115 podStartE2EDuration="4.914515115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:00:23.912264947 +0000 UTC m=+112.068785244" watchObservedRunningTime="2026-09-18 14:00:23.914515115 +0000 UTC m=+112.071035396"
Sep 18 14:00:24.854522 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:24.854461    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-nn5ds"
Sep 18 14:00:24.866913 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:24.866879    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-nn5ds"
Sep 18 14:00:29.447937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:29.447895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:29.448410 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:29.447978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:29.453688 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:29.453663    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:29.880644 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:29.880567    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:00:29.932023 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:29.931983    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54955bb4c4-69dzf"]
Sep 18 14:00:32.020568 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.020533    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:32.378421 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.378350    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-758d69797b-d24nl" podUID="ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" containerName="registry" containerID="cri-o://65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd" gracePeriod=30
Sep 18 14:00:32.651360 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.651303    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 14:00:32.706946 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.706923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-trusted-ca\") pod \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") "
Sep 18 14:00:32.707083 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.706959    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-ca-trust-extracted\") pod \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") "
Sep 18 14:00:32.707083 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.706978    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-bound-sa-token\") pod \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") "
Sep 18 14:00:32.707083 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.707022    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-certificates\") pod \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") "
Sep 18 14:00:32.707083 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.707046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-installation-pull-secrets\") pod \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") "
Sep 18 14:00:32.707083 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.707065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-image-registry-private-configuration\") pod \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") "
Sep 18 14:00:32.707348 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.707103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkjzm\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-kube-api-access-kkjzm\") pod \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") "
Sep 18 14:00:32.707348 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.707266    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls\") pod \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\" (UID: \"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60\") "
Sep 18 14:00:32.707456 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.707397    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:00:32.709947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.709879    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:00:32.711763 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.711732    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-certificates\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:32.711763 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.711762    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-trusted-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:32.711990 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.711945    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:00:32.712103 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.712004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:00:32.712294 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.712268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:00:32.713546 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.713517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-kube-api-access-kkjzm" (OuterVolumeSpecName: "kube-api-access-kkjzm") pod "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60"). InnerVolumeSpecName "kube-api-access-kkjzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:00:32.714012 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.713985    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:00:32.719243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.719220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" (UID: "ee72d2ae-3ac8-418d-b0a6-3a45e0121d60"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:00:32.812249 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.812230    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-installation-pull-secrets\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:32.812249 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.812250    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-image-registry-private-configuration\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:32.812362 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.812260    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkjzm\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-kube-api-access-kkjzm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:32.812362 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.812269    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-registry-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:32.812362 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.812277    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-ca-trust-extracted\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:32.812362 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.812286    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60-bound-sa-token\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:32.896313 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.896284    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" containerID="65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd" exitCode=0
Sep 18 14:00:32.896437 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.896327    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758d69797b-d24nl"
Sep 18 14:00:32.896437 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.896372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758d69797b-d24nl" event={"ID":"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60","Type":"ContainerDied","Data":"65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd"}
Sep 18 14:00:32.896437 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.896431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758d69797b-d24nl" event={"ID":"ee72d2ae-3ac8-418d-b0a6-3a45e0121d60","Type":"ContainerDied","Data":"c4efa3b9ef3b5b03b3aecaa6dfb2be64b15b3ffd4efc923e190980bd5a19f56b"}
Sep 18 14:00:32.896571 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.896460    2563 scope.go:117] "RemoveContainer" containerID="65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd"
Sep 18 14:00:32.905901 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.905880    2563 scope.go:117] "RemoveContainer" containerID="65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd"
Sep 18 14:00:32.906192 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:00:32.906172    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd\": container with ID starting with 65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd not found: ID does not exist" containerID="65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd"
Sep 18 14:00:32.906252 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.906201    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd"} err="failed to get container status \"65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd\": rpc error: code = NotFound desc = could not find container \"65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd\": container with ID starting with 65af7340f11c88247ee1a48c429547b268236da24dcfe23c4f273e5d740da0cd not found: ID does not exist"
Sep 18 14:00:32.917096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.917071    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-758d69797b-d24nl"]
Sep 18 14:00:32.921069 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:32.921051    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-758d69797b-d24nl"]
Sep 18 14:00:34.292722 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:34.292685    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" path="/var/lib/kubelet/pods/ee72d2ae-3ac8-418d-b0a6-3a45e0121d60/volumes"
Sep 18 14:00:36.918872 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:36.918840    2563 generic.go:356] "Generic (PLEG): container finished" podID="aad65a15-66da-43c3-98e9-d46a8a109144" containerID="4ef4d291332a5cd9409e5c2d1018061a78be8eba5f3becf06508856ab87fde16" exitCode=0
Sep 18 14:00:36.919269 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:36.918889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4" event={"ID":"aad65a15-66da-43c3-98e9-d46a8a109144","Type":"ContainerDied","Data":"4ef4d291332a5cd9409e5c2d1018061a78be8eba5f3becf06508856ab87fde16"}
Sep 18 14:00:36.919319 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:36.919266    2563 scope.go:117] "RemoveContainer" containerID="4ef4d291332a5cd9409e5c2d1018061a78be8eba5f3becf06508856ab87fde16"
Sep 18 14:00:37.925564 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:37.925531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-jslm4" event={"ID":"aad65a15-66da-43c3-98e9-d46a8a109144","Type":"ContainerStarted","Data":"d884845a30c0b4a376733d2296abb4b4a3375401461dcceeef62c8e17c3fe423"}
Sep 18 14:00:38.931738 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:38.931706    2563 generic.go:356] "Generic (PLEG): container finished" podID="329d7a0f-ec4f-4aad-842c-7b31e1fd69d4" containerID="9c269d7178e98c83c933ae7702f6c884c3e69de58c4159939655c39b062a7125" exitCode=0
Sep 18 14:00:38.931738 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:38.931741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb" event={"ID":"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4","Type":"ContainerDied","Data":"9c269d7178e98c83c933ae7702f6c884c3e69de58c4159939655c39b062a7125"}
Sep 18 14:00:38.932313 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:38.932027    2563 scope.go:117] "RemoveContainer" containerID="9c269d7178e98c83c933ae7702f6c884c3e69de58c4159939655c39b062a7125"
Sep 18 14:00:39.938650 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:39.938615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-pmhhb" event={"ID":"329d7a0f-ec4f-4aad-842c-7b31e1fd69d4","Type":"ContainerStarted","Data":"5e65976c8a70a57f16379bb003cd3e53971a60ee80bc018849d484f3a05e20c9"}
Sep 18 14:00:54.959987 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:54.959910    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54955bb4c4-69dzf" podUID="b770194f-27a5-4721-a18d-f0ff4633f808" containerName="console" containerID="cri-o://8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef" gracePeriod=15
Sep 18 14:00:55.218065 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.218010    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:55.380350 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.380323    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-oauth-serving-cert\") pod \"b770194f-27a5-4721-a18d-f0ff4633f808\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") "
Sep 18 14:00:55.380350 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.380351    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-serving-cert\") pod \"b770194f-27a5-4721-a18d-f0ff4633f808\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") "
Sep 18 14:00:55.380540 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.380376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-oauth-config\") pod \"b770194f-27a5-4721-a18d-f0ff4633f808\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") "
Sep 18 14:00:55.380540 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.380415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-service-ca\") pod \"b770194f-27a5-4721-a18d-f0ff4633f808\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") "
Sep 18 14:00:55.380540 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.380432    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-console-config\") pod \"b770194f-27a5-4721-a18d-f0ff4633f808\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") "
Sep 18 14:00:55.380540 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.380476    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sb59\" (UniqueName: \"kubernetes.io/projected/b770194f-27a5-4721-a18d-f0ff4633f808-kube-api-access-7sb59\") pod \"b770194f-27a5-4721-a18d-f0ff4633f808\" (UID: \"b770194f-27a5-4721-a18d-f0ff4633f808\") "
Sep 18 14:00:55.380858 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.380825    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b770194f-27a5-4721-a18d-f0ff4633f808" (UID: "b770194f-27a5-4721-a18d-f0ff4633f808"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:00:55.380858 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.380842    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-service-ca" (OuterVolumeSpecName: "service-ca") pod "b770194f-27a5-4721-a18d-f0ff4633f808" (UID: "b770194f-27a5-4721-a18d-f0ff4633f808"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:00:55.381002 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.380853    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-console-config" (OuterVolumeSpecName: "console-config") pod "b770194f-27a5-4721-a18d-f0ff4633f808" (UID: "b770194f-27a5-4721-a18d-f0ff4633f808"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:00:55.382645 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.382622    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b770194f-27a5-4721-a18d-f0ff4633f808" (UID: "b770194f-27a5-4721-a18d-f0ff4633f808"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:00:55.382733 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.382661    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b770194f-27a5-4721-a18d-f0ff4633f808" (UID: "b770194f-27a5-4721-a18d-f0ff4633f808"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:00:55.382733 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.382685    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b770194f-27a5-4721-a18d-f0ff4633f808-kube-api-access-7sb59" (OuterVolumeSpecName: "kube-api-access-7sb59") pod "b770194f-27a5-4721-a18d-f0ff4633f808" (UID: "b770194f-27a5-4721-a18d-f0ff4633f808"). InnerVolumeSpecName "kube-api-access-7sb59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:00:55.481160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.481112    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-console-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:55.481160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.481150    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sb59\" (UniqueName: \"kubernetes.io/projected/b770194f-27a5-4721-a18d-f0ff4633f808-kube-api-access-7sb59\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:55.481160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.481160    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-oauth-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:55.481335 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.481169    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:55.481335 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.481178    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b770194f-27a5-4721-a18d-f0ff4633f808-console-oauth-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:55.481335 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:55.481188    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b770194f-27a5-4721-a18d-f0ff4633f808-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:00:56.010465 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:56.010434    2563 generic.go:356] "Generic (PLEG): container finished" podID="b770194f-27a5-4721-a18d-f0ff4633f808" containerID="8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef" exitCode=0
Sep 18 14:00:56.010831 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:56.010481    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54955bb4c4-69dzf"
Sep 18 14:00:56.010831 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:56.010522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54955bb4c4-69dzf" event={"ID":"b770194f-27a5-4721-a18d-f0ff4633f808","Type":"ContainerDied","Data":"8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef"}
Sep 18 14:00:56.010831 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:56.010576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54955bb4c4-69dzf" event={"ID":"b770194f-27a5-4721-a18d-f0ff4633f808","Type":"ContainerDied","Data":"b817bc1c9e4469a2e7eafda0549d9c6333db8224bed2cd0c6d14418773ad61f9"}
Sep 18 14:00:56.010831 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:56.010602    2563 scope.go:117] "RemoveContainer" containerID="8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef"
Sep 18 14:00:56.019597 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:56.019580    2563 scope.go:117] "RemoveContainer" containerID="8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef"
Sep 18 14:00:56.019847 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:00:56.019803    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef\": container with ID starting with 8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef not found: ID does not exist" containerID="8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef"
Sep 18 14:00:56.019941 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:56.019854    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef"} err="failed to get container status \"8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef\": rpc error: code = NotFound desc = could not find container \"8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef\": container with ID starting with 8c80c7a2ba00e7f55b0bc4b91137da864ca93b6f90b75a72ac8335746e2db0ef not found: ID does not exist"
Sep 18 14:00:56.036411 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:56.036391    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54955bb4c4-69dzf"]
Sep 18 14:00:56.039792 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:56.039775    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54955bb4c4-69dzf"]
Sep 18 14:00:56.287054 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:00:56.287000    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b770194f-27a5-4721-a18d-f0ff4633f808" path="/var/lib/kubelet/pods/b770194f-27a5-4721-a18d-f0ff4633f808/volumes"
Sep 18 14:01:19.768948 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.768914    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77cbb6bc9b-th7kz"]
Sep 18 14:01:19.769389 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.769249    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" containerName="registry"
Sep 18 14:01:19.769389 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.769262    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" containerName="registry"
Sep 18 14:01:19.769389 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.769270    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b770194f-27a5-4721-a18d-f0ff4633f808" containerName="console"
Sep 18 14:01:19.769389 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.769275    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b770194f-27a5-4721-a18d-f0ff4633f808" containerName="console"
Sep 18 14:01:19.769389 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.769332    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b770194f-27a5-4721-a18d-f0ff4633f808" containerName="console"
Sep 18 14:01:19.769389 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.769343    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee72d2ae-3ac8-418d-b0a6-3a45e0121d60" containerName="registry"
Sep 18 14:01:19.772067 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.772042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.785390 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.785366    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77cbb6bc9b-th7kz"]
Sep 18 14:01:19.860687 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.860650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-oauth-serving-cert\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.860833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.860698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-oauth-config\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.860833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.860722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-service-ca\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.860833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.860748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn9mz\" (UniqueName: \"kubernetes.io/projected/61b19e99-12bc-4c25-8dac-42f9b587dfa0-kube-api-access-bn9mz\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.860833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.860809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-config\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.860989 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.860850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-trusted-ca-bundle\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.860989 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.860880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-serving-cert\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.961735 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.961697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-serving-cert\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.961917 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.961757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-oauth-serving-cert\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.961917 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.961779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-oauth-config\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.961917 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.961808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-service-ca\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.961917 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.961838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn9mz\" (UniqueName: \"kubernetes.io/projected/61b19e99-12bc-4c25-8dac-42f9b587dfa0-kube-api-access-bn9mz\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.961917 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.961873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-config\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.961917 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.961912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-trusted-ca-bundle\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.962695 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.962653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-oauth-serving-cert\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.962807 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.962653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-service-ca\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.962807 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.962664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-config\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.963161 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.963117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-trusted-ca-bundle\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.964341 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.964320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-serving-cert\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.964426 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.964327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-oauth-config\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:19.970713 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:19.970688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn9mz\" (UniqueName: \"kubernetes.io/projected/61b19e99-12bc-4c25-8dac-42f9b587dfa0-kube-api-access-bn9mz\") pod \"console-77cbb6bc9b-th7kz\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") " pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:20.081673 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:20.081605    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:20.351196 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:20.351170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77cbb6bc9b-th7kz"]
Sep 18 14:01:20.356044 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:01:20.356008    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b19e99_12bc_4c25_8dac_42f9b587dfa0.slice/crio-b3b975abdb81481364309cd1b5fc8768dca848b500bfe6e8aa3a5d5c9ca78436 WatchSource:0}: Error finding container b3b975abdb81481364309cd1b5fc8768dca848b500bfe6e8aa3a5d5c9ca78436: Status 404 returned error can't find the container with id b3b975abdb81481364309cd1b5fc8768dca848b500bfe6e8aa3a5d5c9ca78436
Sep 18 14:01:21.114828 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:21.114793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77cbb6bc9b-th7kz" event={"ID":"61b19e99-12bc-4c25-8dac-42f9b587dfa0","Type":"ContainerStarted","Data":"6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc"}
Sep 18 14:01:21.115222 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:21.114842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77cbb6bc9b-th7kz" event={"ID":"61b19e99-12bc-4c25-8dac-42f9b587dfa0","Type":"ContainerStarted","Data":"b3b975abdb81481364309cd1b5fc8768dca848b500bfe6e8aa3a5d5c9ca78436"}
Sep 18 14:01:21.133238 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:21.133189    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77cbb6bc9b-th7kz" podStartSLOduration=2.133175087 podStartE2EDuration="2.133175087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:01:21.132262769 +0000 UTC m=+169.288783064" watchObservedRunningTime="2026-09-18 14:01:21.133175087 +0000 UTC m=+169.289695368"
Sep 18 14:01:27.157084 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.157044    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77cbb6bc9b-th7kz"]
Sep 18 14:01:27.226043 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.226001    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f7bbbb964-hx9pt"]
Sep 18 14:01:27.228000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.227977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.265393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.265358    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f7bbbb964-hx9pt"]
Sep 18 14:01:27.312191 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.312161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-trusted-ca-bundle\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.312318 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.312197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-oauth-config\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.312318 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.312218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-service-ca\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.312318 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.312267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-serving-cert\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.312318 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.312280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-console-config\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.312318 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.312298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvwn\" (UniqueName: \"kubernetes.io/projected/f4feab4f-fead-4471-97da-6393e85ad441-kube-api-access-9jvwn\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.312543 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.312378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-oauth-serving-cert\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.413149 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.413072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-service-ca\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.413149 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.413109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-serving-cert\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.413149 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.413125    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-console-config\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.413343 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.413178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvwn\" (UniqueName: \"kubernetes.io/projected/f4feab4f-fead-4471-97da-6393e85ad441-kube-api-access-9jvwn\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.413343 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.413197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-oauth-serving-cert\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.413343 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.413232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-trusted-ca-bundle\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.413343 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.413254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-oauth-config\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.413947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.413922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-console-config\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.414044 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.413922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-service-ca\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.414044 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.413982    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-oauth-serving-cert\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.414175 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.414088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-trusted-ca-bundle\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.415709 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.415687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-oauth-config\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.415877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.415858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-serving-cert\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.421353 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.421334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvwn\" (UniqueName: \"kubernetes.io/projected/f4feab4f-fead-4471-97da-6393e85ad441-kube-api-access-9jvwn\") pod \"console-6f7bbbb964-hx9pt\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") " pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.537505 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.537464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:27.816310 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:01:27.816277    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4feab4f_fead_4471_97da_6393e85ad441.slice/crio-b06260c0d1119e545629ebb98204d97a33acbcd71aba4efccaa092971c1a0f4a WatchSource:0}: Error finding container b06260c0d1119e545629ebb98204d97a33acbcd71aba4efccaa092971c1a0f4a: Status 404 returned error can't find the container with id b06260c0d1119e545629ebb98204d97a33acbcd71aba4efccaa092971c1a0f4a
Sep 18 14:01:27.816552 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:27.816535    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f7bbbb964-hx9pt"]
Sep 18 14:01:28.145826 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:28.145741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7bbbb964-hx9pt" event={"ID":"f4feab4f-fead-4471-97da-6393e85ad441","Type":"ContainerStarted","Data":"b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a"}
Sep 18 14:01:28.145826 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:28.145789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7bbbb964-hx9pt" event={"ID":"f4feab4f-fead-4471-97da-6393e85ad441","Type":"ContainerStarted","Data":"b06260c0d1119e545629ebb98204d97a33acbcd71aba4efccaa092971c1a0f4a"}
Sep 18 14:01:28.177767 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:28.177723    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f7bbbb964-hx9pt" podStartSLOduration=1.177709884 podStartE2EDuration="1.177709884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:01:28.177410631 +0000 UTC m=+176.333930913" watchObservedRunningTime="2026-09-18 14:01:28.177709884 +0000 UTC m=+176.334230237"
Sep 18 14:01:30.082100 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:30.082051    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:37.537601 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:37.537561    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:37.538021 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:37.537667    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:37.542289 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:37.542267    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:38.192283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:38.192251    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:01:38.259839 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:38.259801    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-94b9b8989-n9tdx"]
Sep 18 14:01:52.178504 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.178454    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77cbb6bc9b-th7kz" podUID="61b19e99-12bc-4c25-8dac-42f9b587dfa0" containerName="console" containerID="cri-o://6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc" gracePeriod=15
Sep 18 14:01:52.419831 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.419808    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:52.496220 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.496151    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-serving-cert\") pod \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") "
Sep 18 14:01:52.496220 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.496201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-trusted-ca-bundle\") pod \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") "
Sep 18 14:01:52.496414 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.496254    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-config\") pod \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") "
Sep 18 14:01:52.496414 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.496273    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-service-ca\") pod \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") "
Sep 18 14:01:52.496414 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.496296    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-oauth-serving-cert\") pod \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") "
Sep 18 14:01:52.496414 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.496322    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-oauth-config\") pod \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") "
Sep 18 14:01:52.496414 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.496352    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn9mz\" (UniqueName: \"kubernetes.io/projected/61b19e99-12bc-4c25-8dac-42f9b587dfa0-kube-api-access-bn9mz\") pod \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\" (UID: \"61b19e99-12bc-4c25-8dac-42f9b587dfa0\") "
Sep 18 14:01:52.496757 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.496664    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-config" (OuterVolumeSpecName: "console-config") pod "61b19e99-12bc-4c25-8dac-42f9b587dfa0" (UID: "61b19e99-12bc-4c25-8dac-42f9b587dfa0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:52.496757 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.496685    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-service-ca" (OuterVolumeSpecName: "service-ca") pod "61b19e99-12bc-4c25-8dac-42f9b587dfa0" (UID: "61b19e99-12bc-4c25-8dac-42f9b587dfa0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:52.496757 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.496702    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "61b19e99-12bc-4c25-8dac-42f9b587dfa0" (UID: "61b19e99-12bc-4c25-8dac-42f9b587dfa0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:52.496757 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.496712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "61b19e99-12bc-4c25-8dac-42f9b587dfa0" (UID: "61b19e99-12bc-4c25-8dac-42f9b587dfa0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:01:52.498300 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.498274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "61b19e99-12bc-4c25-8dac-42f9b587dfa0" (UID: "61b19e99-12bc-4c25-8dac-42f9b587dfa0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:01:52.498548 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.498521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "61b19e99-12bc-4c25-8dac-42f9b587dfa0" (UID: "61b19e99-12bc-4c25-8dac-42f9b587dfa0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:01:52.498599 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.498546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b19e99-12bc-4c25-8dac-42f9b587dfa0-kube-api-access-bn9mz" (OuterVolumeSpecName: "kube-api-access-bn9mz") pod "61b19e99-12bc-4c25-8dac-42f9b587dfa0" (UID: "61b19e99-12bc-4c25-8dac-42f9b587dfa0"). InnerVolumeSpecName "kube-api-access-bn9mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:01:52.597264 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.597228    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:01:52.597264 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.597268    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-oauth-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:01:52.597430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.597282    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-oauth-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:01:52.597430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.597294    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn9mz\" (UniqueName: \"kubernetes.io/projected/61b19e99-12bc-4c25-8dac-42f9b587dfa0-kube-api-access-bn9mz\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:01:52.597430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.597307    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:01:52.597430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.597319    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-trusted-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:01:52.597430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:52.597331    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61b19e99-12bc-4c25-8dac-42f9b587dfa0-console-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:01:53.256787 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:53.256748    2563 generic.go:356] "Generic (PLEG): container finished" podID="61b19e99-12bc-4c25-8dac-42f9b587dfa0" containerID="6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc" exitCode=0
Sep 18 14:01:53.257309 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:53.256803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77cbb6bc9b-th7kz" event={"ID":"61b19e99-12bc-4c25-8dac-42f9b587dfa0","Type":"ContainerDied","Data":"6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc"}
Sep 18 14:01:53.257309 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:53.256809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77cbb6bc9b-th7kz"
Sep 18 14:01:53.257309 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:53.256848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77cbb6bc9b-th7kz" event={"ID":"61b19e99-12bc-4c25-8dac-42f9b587dfa0","Type":"ContainerDied","Data":"b3b975abdb81481364309cd1b5fc8768dca848b500bfe6e8aa3a5d5c9ca78436"}
Sep 18 14:01:53.257309 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:53.256871    2563 scope.go:117] "RemoveContainer" containerID="6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc"
Sep 18 14:01:53.266638 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:53.266616    2563 scope.go:117] "RemoveContainer" containerID="6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc"
Sep 18 14:01:53.266944 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:01:53.266923    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc\": container with ID starting with 6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc not found: ID does not exist" containerID="6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc"
Sep 18 14:01:53.266987 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:53.266953    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc"} err="failed to get container status \"6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc\": rpc error: code = NotFound desc = could not find container \"6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc\": container with ID starting with 6c3371980943bbe1720ff6a4cfcb1e4af92bd1d270367b911ead2d3a16c47fbc not found: ID does not exist"
Sep 18 14:01:53.278973 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:53.278943    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77cbb6bc9b-th7kz"]
Sep 18 14:01:53.283150 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:53.283110    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77cbb6bc9b-th7kz"]
Sep 18 14:01:54.286217 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:01:54.286174    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61b19e99-12bc-4c25-8dac-42f9b587dfa0" path="/var/lib/kubelet/pods/61b19e99-12bc-4c25-8dac-42f9b587dfa0/volumes"
Sep 18 14:02:03.280443 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.280388    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-94b9b8989-n9tdx" podUID="8ef57b54-12fe-4b31-a658-d84fca154106" containerName="console" containerID="cri-o://6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99" gracePeriod=15
Sep 18 14:02:03.531876 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.531821    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:02:03.680322 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.680297    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-service-ca\") pod \"8ef57b54-12fe-4b31-a658-d84fca154106\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") "
Sep 18 14:02:03.680444 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.680327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv6jj\" (UniqueName: \"kubernetes.io/projected/8ef57b54-12fe-4b31-a658-d84fca154106-kube-api-access-pv6jj\") pod \"8ef57b54-12fe-4b31-a658-d84fca154106\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") "
Sep 18 14:02:03.680444 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.680352    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-oauth-serving-cert\") pod \"8ef57b54-12fe-4b31-a658-d84fca154106\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") "
Sep 18 14:02:03.680444 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.680383    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-console-config\") pod \"8ef57b54-12fe-4b31-a658-d84fca154106\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") "
Sep 18 14:02:03.680444 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.680415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-oauth-config\") pod \"8ef57b54-12fe-4b31-a658-d84fca154106\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") "
Sep 18 14:02:03.680638 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.680453    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-serving-cert\") pod \"8ef57b54-12fe-4b31-a658-d84fca154106\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") "
Sep 18 14:02:03.680638 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.680516    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-trusted-ca-bundle\") pod \"8ef57b54-12fe-4b31-a658-d84fca154106\" (UID: \"8ef57b54-12fe-4b31-a658-d84fca154106\") "
Sep 18 14:02:03.680812 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.680783    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8ef57b54-12fe-4b31-a658-d84fca154106" (UID: "8ef57b54-12fe-4b31-a658-d84fca154106"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:02:03.680873 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.680847    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-console-config" (OuterVolumeSpecName: "console-config") pod "8ef57b54-12fe-4b31-a658-d84fca154106" (UID: "8ef57b54-12fe-4b31-a658-d84fca154106"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:02:03.681009 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.680990    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8ef57b54-12fe-4b31-a658-d84fca154106" (UID: "8ef57b54-12fe-4b31-a658-d84fca154106"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:02:03.681117 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.681095    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-service-ca" (OuterVolumeSpecName: "service-ca") pod "8ef57b54-12fe-4b31-a658-d84fca154106" (UID: "8ef57b54-12fe-4b31-a658-d84fca154106"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:02:03.682624 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.682596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8ef57b54-12fe-4b31-a658-d84fca154106" (UID: "8ef57b54-12fe-4b31-a658-d84fca154106"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:02:03.682624 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.682607    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef57b54-12fe-4b31-a658-d84fca154106-kube-api-access-pv6jj" (OuterVolumeSpecName: "kube-api-access-pv6jj") pod "8ef57b54-12fe-4b31-a658-d84fca154106" (UID: "8ef57b54-12fe-4b31-a658-d84fca154106"). InnerVolumeSpecName "kube-api-access-pv6jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:03.682757 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.682674    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8ef57b54-12fe-4b31-a658-d84fca154106" (UID: "8ef57b54-12fe-4b31-a658-d84fca154106"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:02:03.780948 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.780926    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:02:03.780948 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.780945    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv6jj\" (UniqueName: \"kubernetes.io/projected/8ef57b54-12fe-4b31-a658-d84fca154106-kube-api-access-pv6jj\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:02:03.781066 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.780956    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-oauth-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:02:03.781066 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.780964    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-console-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:02:03.781066 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.780974    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-oauth-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:02:03.781066 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.780982    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ef57b54-12fe-4b31-a658-d84fca154106-console-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:02:03.781066 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:03.780990    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ef57b54-12fe-4b31-a658-d84fca154106-trusted-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:02:04.305203 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:04.305169    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ef57b54-12fe-4b31-a658-d84fca154106" containerID="6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99" exitCode=0
Sep 18 14:02:04.305515 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:04.305218    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-94b9b8989-n9tdx"
Sep 18 14:02:04.305515 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:04.305249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94b9b8989-n9tdx" event={"ID":"8ef57b54-12fe-4b31-a658-d84fca154106","Type":"ContainerDied","Data":"6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99"}
Sep 18 14:02:04.305515 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:04.305312    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-94b9b8989-n9tdx" event={"ID":"8ef57b54-12fe-4b31-a658-d84fca154106","Type":"ContainerDied","Data":"de8289ee8c0876e157315203a71d80fd4abf7e02066e62ee44f9f43acf132a49"}
Sep 18 14:02:04.305515 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:04.305340    2563 scope.go:117] "RemoveContainer" containerID="6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99"
Sep 18 14:02:04.314048 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:04.314026    2563 scope.go:117] "RemoveContainer" containerID="6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99"
Sep 18 14:02:04.314355 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:02:04.314334    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99\": container with ID starting with 6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99 not found: ID does not exist" containerID="6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99"
Sep 18 14:02:04.314426 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:04.314368    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99"} err="failed to get container status \"6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99\": rpc error: code = NotFound desc = could not find container \"6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99\": container with ID starting with 6a8843f22d8edd9d486bde766a613f71ad120ad13349d8cd7a72036356508a99 not found: ID does not exist"
Sep 18 14:02:04.322019 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:04.321995    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-94b9b8989-n9tdx"]
Sep 18 14:02:04.326056 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:04.326037    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-94b9b8989-n9tdx"]
Sep 18 14:02:06.286756 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:06.286718    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ef57b54-12fe-4b31-a658-d84fca154106" path="/var/lib/kubelet/pods/8ef57b54-12fe-4b31-a658-d84fca154106/volumes"
Sep 18 14:02:29.312544 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.312505    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"]
Sep 18 14:02:29.313000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.312795    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61b19e99-12bc-4c25-8dac-42f9b587dfa0" containerName="console"
Sep 18 14:02:29.313000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.312807    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="61b19e99-12bc-4c25-8dac-42f9b587dfa0" containerName="console"
Sep 18 14:02:29.313000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.312828    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ef57b54-12fe-4b31-a658-d84fca154106" containerName="console"
Sep 18 14:02:29.313000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.312833    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ef57b54-12fe-4b31-a658-d84fca154106" containerName="console"
Sep 18 14:02:29.313000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.312888    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="61b19e99-12bc-4c25-8dac-42f9b587dfa0" containerName="console"
Sep 18 14:02:29.313000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.312895    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ef57b54-12fe-4b31-a658-d84fca154106" containerName="console"
Sep 18 14:02:29.314778 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.314761    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:29.317362 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.317333    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4tjp\""
Sep 18 14:02:29.317509 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.317492    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 14:02:29.318145 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.318114    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 14:02:29.328363 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.328336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"]
Sep 18 14:02:29.457818 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.457790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgkt\" (UniqueName: \"kubernetes.io/projected/b25ed33a-6f59-4d7c-b16f-91b670200106-kube-api-access-twgkt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:29.457947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.457840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:29.457947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.457924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:29.558648 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.558605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:29.558772 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.558663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgkt\" (UniqueName: \"kubernetes.io/projected/b25ed33a-6f59-4d7c-b16f-91b670200106-kube-api-access-twgkt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:29.558772 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.558706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:29.558997 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.558963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:29.559068 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.559040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:29.567373 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.567320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgkt\" (UniqueName: \"kubernetes.io/projected/b25ed33a-6f59-4d7c-b16f-91b670200106-kube-api-access-twgkt\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:29.623881 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.623858    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:29.892754 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:29.892727    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"]
Sep 18 14:02:30.424053 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:30.424013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89" event={"ID":"b25ed33a-6f59-4d7c-b16f-91b670200106","Type":"ContainerStarted","Data":"b1203005038b391977f91090b776267728c50f041840e04dcfacce5fbf24c35f"}
Sep 18 14:02:36.449028 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:36.448989    2563 generic.go:356] "Generic (PLEG): container finished" podID="b25ed33a-6f59-4d7c-b16f-91b670200106" containerID="5557aae4cc0a66f141cc0e0466931f51c0341f67615150d1bb748e5c11ef137a" exitCode=0
Sep 18 14:02:36.449435 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:36.449032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89" event={"ID":"b25ed33a-6f59-4d7c-b16f-91b670200106","Type":"ContainerDied","Data":"5557aae4cc0a66f141cc0e0466931f51c0341f67615150d1bb748e5c11ef137a"}
Sep 18 14:02:38.460607 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:38.460571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89" event={"ID":"b25ed33a-6f59-4d7c-b16f-91b670200106","Type":"ContainerStarted","Data":"980c85895ac4e181f5f5c3782313e114721869f97816cbdbe92d27959094bcde"}
Sep 18 14:02:39.465728 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:39.465697    2563 generic.go:356] "Generic (PLEG): container finished" podID="b25ed33a-6f59-4d7c-b16f-91b670200106" containerID="980c85895ac4e181f5f5c3782313e114721869f97816cbdbe92d27959094bcde" exitCode=0
Sep 18 14:02:39.466087 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:39.465765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89" event={"ID":"b25ed33a-6f59-4d7c-b16f-91b670200106","Type":"ContainerDied","Data":"980c85895ac4e181f5f5c3782313e114721869f97816cbdbe92d27959094bcde"}
Sep 18 14:02:45.493654 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:45.493612    2563 generic.go:356] "Generic (PLEG): container finished" podID="b25ed33a-6f59-4d7c-b16f-91b670200106" containerID="e390d9921aad18bcc8b24bfd5104fded26e8300cf1742b7930da832596fcf05f" exitCode=0
Sep 18 14:02:45.494105 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:45.493698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89" event={"ID":"b25ed33a-6f59-4d7c-b16f-91b670200106","Type":"ContainerDied","Data":"e390d9921aad18bcc8b24bfd5104fded26e8300cf1742b7930da832596fcf05f"}
Sep 18 14:02:46.623202 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:46.623179    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:46.693920 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:46.693896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-bundle\") pod \"b25ed33a-6f59-4d7c-b16f-91b670200106\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") "
Sep 18 14:02:46.694033 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:46.693949    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-util\") pod \"b25ed33a-6f59-4d7c-b16f-91b670200106\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") "
Sep 18 14:02:46.694033 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:46.693979    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twgkt\" (UniqueName: \"kubernetes.io/projected/b25ed33a-6f59-4d7c-b16f-91b670200106-kube-api-access-twgkt\") pod \"b25ed33a-6f59-4d7c-b16f-91b670200106\" (UID: \"b25ed33a-6f59-4d7c-b16f-91b670200106\") "
Sep 18 14:02:46.694539 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:46.694509    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-bundle" (OuterVolumeSpecName: "bundle") pod "b25ed33a-6f59-4d7c-b16f-91b670200106" (UID: "b25ed33a-6f59-4d7c-b16f-91b670200106"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:02:46.696068 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:46.696042    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25ed33a-6f59-4d7c-b16f-91b670200106-kube-api-access-twgkt" (OuterVolumeSpecName: "kube-api-access-twgkt") pod "b25ed33a-6f59-4d7c-b16f-91b670200106" (UID: "b25ed33a-6f59-4d7c-b16f-91b670200106"). InnerVolumeSpecName "kube-api-access-twgkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:02:46.701266 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:46.701226    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-util" (OuterVolumeSpecName: "util") pod "b25ed33a-6f59-4d7c-b16f-91b670200106" (UID: "b25ed33a-6f59-4d7c-b16f-91b670200106"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:02:46.794893 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:46.794836    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:02:46.794893 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:46.794857    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b25ed33a-6f59-4d7c-b16f-91b670200106-util\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:02:46.794893 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:46.794866    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twgkt\" (UniqueName: \"kubernetes.io/projected/b25ed33a-6f59-4d7c-b16f-91b670200106-kube-api-access-twgkt\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:02:47.504506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:47.504464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89" event={"ID":"b25ed33a-6f59-4d7c-b16f-91b670200106","Type":"ContainerDied","Data":"b1203005038b391977f91090b776267728c50f041840e04dcfacce5fbf24c35f"}
Sep 18 14:02:47.504506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:47.504502    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vmn89"
Sep 18 14:02:47.504717 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:47.504517    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1203005038b391977f91090b776267728c50f041840e04dcfacce5fbf24c35f"
Sep 18 14:02:52.588777 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.588732    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28"]
Sep 18 14:02:52.589294 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.589220    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25ed33a-6f59-4d7c-b16f-91b670200106" containerName="util"
Sep 18 14:02:52.589294 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.589239    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25ed33a-6f59-4d7c-b16f-91b670200106" containerName="util"
Sep 18 14:02:52.589294 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.589254    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25ed33a-6f59-4d7c-b16f-91b670200106" containerName="pull"
Sep 18 14:02:52.589294 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.589262    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25ed33a-6f59-4d7c-b16f-91b670200106" containerName="pull"
Sep 18 14:02:52.589294 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.589272    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25ed33a-6f59-4d7c-b16f-91b670200106" containerName="extract"
Sep 18 14:02:52.589294 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.589280    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25ed33a-6f59-4d7c-b16f-91b670200106" containerName="extract"
Sep 18 14:02:52.589581 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.589362    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25ed33a-6f59-4d7c-b16f-91b670200106" containerName="extract"
Sep 18 14:02:52.592493 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.592468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28"
Sep 18 14:02:52.594798 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.594775    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 18 14:02:52.594920 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.594849    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 18 14:02:52.594974 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.594931    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zj9kl\""
Sep 18 14:02:52.606479 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.606442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28"]
Sep 18 14:02:52.635877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.635851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acaf6a36-3332-4949-8933-c8f8a46a360b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sgx28\" (UID: \"acaf6a36-3332-4949-8933-c8f8a46a360b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28"
Sep 18 14:02:52.635985 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.635904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb86m\" (UniqueName: \"kubernetes.io/projected/acaf6a36-3332-4949-8933-c8f8a46a360b-kube-api-access-sb86m\") pod \"cert-manager-operator-controller-manager-55c444bf86-sgx28\" (UID: \"acaf6a36-3332-4949-8933-c8f8a46a360b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28"
Sep 18 14:02:52.736727 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.736703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb86m\" (UniqueName: \"kubernetes.io/projected/acaf6a36-3332-4949-8933-c8f8a46a360b-kube-api-access-sb86m\") pod \"cert-manager-operator-controller-manager-55c444bf86-sgx28\" (UID: \"acaf6a36-3332-4949-8933-c8f8a46a360b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28"
Sep 18 14:02:52.736833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.736750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acaf6a36-3332-4949-8933-c8f8a46a360b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sgx28\" (UID: \"acaf6a36-3332-4949-8933-c8f8a46a360b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28"
Sep 18 14:02:52.737093 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.737077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acaf6a36-3332-4949-8933-c8f8a46a360b-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-sgx28\" (UID: \"acaf6a36-3332-4949-8933-c8f8a46a360b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28"
Sep 18 14:02:52.747962 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.747935    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb86m\" (UniqueName: \"kubernetes.io/projected/acaf6a36-3332-4949-8933-c8f8a46a360b-kube-api-access-sb86m\") pod \"cert-manager-operator-controller-manager-55c444bf86-sgx28\" (UID: \"acaf6a36-3332-4949-8933-c8f8a46a360b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28"
Sep 18 14:02:52.902451 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:52.902386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28"
Sep 18 14:02:53.178939 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:53.178912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28"]
Sep 18 14:02:53.182176 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:02:53.182149    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacaf6a36_3332_4949_8933_c8f8a46a360b.slice/crio-1915ace7810a1795c246ad9ef3d75fd8d701983df11114f980de4f0577c4d211 WatchSource:0}: Error finding container 1915ace7810a1795c246ad9ef3d75fd8d701983df11114f980de4f0577c4d211: Status 404 returned error can't find the container with id 1915ace7810a1795c246ad9ef3d75fd8d701983df11114f980de4f0577c4d211
Sep 18 14:02:53.531995 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:53.531926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28" event={"ID":"acaf6a36-3332-4949-8933-c8f8a46a360b","Type":"ContainerStarted","Data":"1915ace7810a1795c246ad9ef3d75fd8d701983df11114f980de4f0577c4d211"}
Sep 18 14:02:55.543659 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:55.543620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28" event={"ID":"acaf6a36-3332-4949-8933-c8f8a46a360b","Type":"ContainerStarted","Data":"0e961371ab6eb0b72dbf3802afa49455bb52f0acc6ef4f64ca3a8e4e253d8628"}
Sep 18 14:02:55.573375 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:55.573328    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-sgx28" podStartSLOduration=2.051655894 podStartE2EDuration="3.573311667s" totalImagesPullingTime="1.521655773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:02:55.573271134 +0000 UTC m=+263.729791414" watchObservedRunningTime="2026-09-18 14:02:55.573311667 +0000 UTC m=+263.729831944"
Sep 18 14:02:56.558212 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.558179    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"]
Sep 18 14:02:56.561846 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.561827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:02:56.565891 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.565869    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4tjp\""
Sep 18 14:02:56.566484 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.566460    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 14:02:56.566582 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.566501    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 14:02:56.586058 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.586039    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"]
Sep 18 14:02:56.666600 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.666575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:02:56.666706 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.666635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:02:56.666706 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.666686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdqr9\" (UniqueName: \"kubernetes.io/projected/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-kube-api-access-pdqr9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:02:56.767519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.767495    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:02:56.767620 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.767562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:02:56.767620 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.767584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdqr9\" (UniqueName: \"kubernetes.io/projected/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-kube-api-access-pdqr9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:02:56.767865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.767848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:02:56.767904 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.767872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:02:56.784694 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.784671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdqr9\" (UniqueName: \"kubernetes.io/projected/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-kube-api-access-pdqr9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:02:56.870752 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:56.870695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:02:57.157626 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:57.157591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"]
Sep 18 14:02:57.159816 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:02:57.159775    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82e95c6_00ac_4ced_8ad7_9d7fe32f03f6.slice/crio-0b9743a1391e912046d73768f468132cf5bdee0ad50344b8e27813637dff29a2 WatchSource:0}: Error finding container 0b9743a1391e912046d73768f468132cf5bdee0ad50344b8e27813637dff29a2: Status 404 returned error can't find the container with id 0b9743a1391e912046d73768f468132cf5bdee0ad50344b8e27813637dff29a2
Sep 18 14:02:57.553484 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:57.553456    2563 generic.go:356] "Generic (PLEG): container finished" podID="c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" containerID="55567e440d700b2db28004650c903a71d05e2707448c7d5797d88260dbfa5a47" exitCode=0
Sep 18 14:02:57.553627 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:57.553545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p" event={"ID":"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6","Type":"ContainerDied","Data":"55567e440d700b2db28004650c903a71d05e2707448c7d5797d88260dbfa5a47"}
Sep 18 14:02:57.553627 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:57.553597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p" event={"ID":"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6","Type":"ContainerStarted","Data":"0b9743a1391e912046d73768f468132cf5bdee0ad50344b8e27813637dff29a2"}
Sep 18 14:02:59.564285 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:02:59.564255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p" event={"ID":"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6","Type":"ContainerStarted","Data":"41aac2a902cfb80fe85d35342c9dcafb02f23cba243873963b37e9dfe47ce675"}
Sep 18 14:03:00.570034 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:00.569989    2563 generic.go:356] "Generic (PLEG): container finished" podID="c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" containerID="41aac2a902cfb80fe85d35342c9dcafb02f23cba243873963b37e9dfe47ce675" exitCode=0
Sep 18 14:03:00.570403 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:00.570071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p" event={"ID":"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6","Type":"ContainerDied","Data":"41aac2a902cfb80fe85d35342c9dcafb02f23cba243873963b37e9dfe47ce675"}
Sep 18 14:03:01.576011 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:01.575974    2563 generic.go:356] "Generic (PLEG): container finished" podID="c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" containerID="dc9bde26e6d090a9f51d47561a45bf11597deed5132b96015ccf08117d19bf55" exitCode=0
Sep 18 14:03:01.576372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:01.576010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p" event={"ID":"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6","Type":"ContainerDied","Data":"dc9bde26e6d090a9f51d47561a45bf11597deed5132b96015ccf08117d19bf55"}
Sep 18 14:03:02.708309 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:02.708286    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:03:02.815204 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:02.815170    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdqr9\" (UniqueName: \"kubernetes.io/projected/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-kube-api-access-pdqr9\") pod \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") "
Sep 18 14:03:02.815323 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:02.815260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-bundle\") pod \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") "
Sep 18 14:03:02.815323 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:02.815277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-util\") pod \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\" (UID: \"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6\") "
Sep 18 14:03:02.815633 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:02.815604    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-bundle" (OuterVolumeSpecName: "bundle") pod "c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" (UID: "c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:03:02.817118 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:02.817092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-kube-api-access-pdqr9" (OuterVolumeSpecName: "kube-api-access-pdqr9") pod "c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" (UID: "c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6"). InnerVolumeSpecName "kube-api-access-pdqr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:03:02.867939 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:02.867888    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-util" (OuterVolumeSpecName: "util") pod "c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" (UID: "c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:03:02.916254 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:02.916234    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:03:02.916254 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:02.916254    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-util\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:03:02.916377 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:02.916262    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdqr9\" (UniqueName: \"kubernetes.io/projected/c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6-kube-api-access-pdqr9\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:03:03.588861 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:03.588820    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p" event={"ID":"c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6","Type":"ContainerDied","Data":"0b9743a1391e912046d73768f468132cf5bdee0ad50344b8e27813637dff29a2"}
Sep 18 14:03:03.589038 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:03.588886    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9743a1391e912046d73768f468132cf5bdee0ad50344b8e27813637dff29a2"
Sep 18 14:03:03.589038 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:03.588832    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftnz4p"
Sep 18 14:03:09.395067 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.395027    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5rdfc"]
Sep 18 14:03:09.395670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.395483    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" containerName="util"
Sep 18 14:03:09.395670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.395509    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" containerName="util"
Sep 18 14:03:09.395670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.395525    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" containerName="pull"
Sep 18 14:03:09.395670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.395534    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" containerName="pull"
Sep 18 14:03:09.395670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.395560    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" containerName="extract"
Sep 18 14:03:09.395670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.395568    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" containerName="extract"
Sep 18 14:03:09.395670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.395640    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c82e95c6-00ac-4ced-8ad7-9d7fe32f03f6" containerName="extract"
Sep 18 14:03:09.400193 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.400173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5rdfc"
Sep 18 14:03:09.402661 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.402640    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 14:03:09.402751 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.402641    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 14:03:09.402751 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.402675    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-b5cm8\""
Sep 18 14:03:09.408867 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.408841    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5rdfc"]
Sep 18 14:03:09.468702 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.468669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a-bound-sa-token\") pod \"cert-manager-55b4d954d5-5rdfc\" (UID: \"a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a\") " pod="cert-manager/cert-manager-55b4d954d5-5rdfc"
Sep 18 14:03:09.468899 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.468707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rpn\" (UniqueName: \"kubernetes.io/projected/a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a-kube-api-access-g5rpn\") pod \"cert-manager-55b4d954d5-5rdfc\" (UID: \"a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a\") " pod="cert-manager/cert-manager-55b4d954d5-5rdfc"
Sep 18 14:03:09.569637 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.569607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a-bound-sa-token\") pod \"cert-manager-55b4d954d5-5rdfc\" (UID: \"a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a\") " pod="cert-manager/cert-manager-55b4d954d5-5rdfc"
Sep 18 14:03:09.569792 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.569645    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rpn\" (UniqueName: \"kubernetes.io/projected/a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a-kube-api-access-g5rpn\") pod \"cert-manager-55b4d954d5-5rdfc\" (UID: \"a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a\") " pod="cert-manager/cert-manager-55b4d954d5-5rdfc"
Sep 18 14:03:09.579383 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.579354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a-bound-sa-token\") pod \"cert-manager-55b4d954d5-5rdfc\" (UID: \"a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a\") " pod="cert-manager/cert-manager-55b4d954d5-5rdfc"
Sep 18 14:03:09.579507 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.579484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rpn\" (UniqueName: \"kubernetes.io/projected/a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a-kube-api-access-g5rpn\") pod \"cert-manager-55b4d954d5-5rdfc\" (UID: \"a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a\") " pod="cert-manager/cert-manager-55b4d954d5-5rdfc"
Sep 18 14:03:09.727607 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:09.727564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5rdfc"
Sep 18 14:03:10.006368 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.006334    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5rdfc"]
Sep 18 14:03:10.010185 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.009868    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg"]
Sep 18 14:03:10.014526 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.014496    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg"
Sep 18 14:03:10.018013 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.017995    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 14:03:10.018100 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.018023    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6grzh\""
Sep 18 14:03:10.018846 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.018833    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 14:03:10.026872 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.026840    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg"]
Sep 18 14:03:10.074691 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.074655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/039e20df-12e8-4f1f-a0aa-a4e79c1c759f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qzhxg\" (UID: \"039e20df-12e8-4f1f-a0aa-a4e79c1c759f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg"
Sep 18 14:03:10.074691 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.074695    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpcww\" (UniqueName: \"kubernetes.io/projected/039e20df-12e8-4f1f-a0aa-a4e79c1c759f-kube-api-access-qpcww\") pod \"openshift-lws-operator-fd8ccff4c-qzhxg\" (UID: \"039e20df-12e8-4f1f-a0aa-a4e79c1c759f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg"
Sep 18 14:03:10.175296 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.175254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpcww\" (UniqueName: \"kubernetes.io/projected/039e20df-12e8-4f1f-a0aa-a4e79c1c759f-kube-api-access-qpcww\") pod \"openshift-lws-operator-fd8ccff4c-qzhxg\" (UID: \"039e20df-12e8-4f1f-a0aa-a4e79c1c759f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg"
Sep 18 14:03:10.175512 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.175372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/039e20df-12e8-4f1f-a0aa-a4e79c1c759f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qzhxg\" (UID: \"039e20df-12e8-4f1f-a0aa-a4e79c1c759f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg"
Sep 18 14:03:10.175760 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.175739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/039e20df-12e8-4f1f-a0aa-a4e79c1c759f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qzhxg\" (UID: \"039e20df-12e8-4f1f-a0aa-a4e79c1c759f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg"
Sep 18 14:03:10.183357 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.183332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpcww\" (UniqueName: \"kubernetes.io/projected/039e20df-12e8-4f1f-a0aa-a4e79c1c759f-kube-api-access-qpcww\") pod \"openshift-lws-operator-fd8ccff4c-qzhxg\" (UID: \"039e20df-12e8-4f1f-a0aa-a4e79c1c759f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg"
Sep 18 14:03:10.324031 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.323930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg"
Sep 18 14:03:10.593365 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.593338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg"]
Sep 18 14:03:10.595762 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:03:10.595719    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod039e20df_12e8_4f1f_a0aa_a4e79c1c759f.slice/crio-f8dddcccde6ab8b91863292bd21a3883d80428dd8fd175d0fbcd3756d5b4d7e1 WatchSource:0}: Error finding container f8dddcccde6ab8b91863292bd21a3883d80428dd8fd175d0fbcd3756d5b4d7e1: Status 404 returned error can't find the container with id f8dddcccde6ab8b91863292bd21a3883d80428dd8fd175d0fbcd3756d5b4d7e1
Sep 18 14:03:10.620050 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.620013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg" event={"ID":"039e20df-12e8-4f1f-a0aa-a4e79c1c759f","Type":"ContainerStarted","Data":"f8dddcccde6ab8b91863292bd21a3883d80428dd8fd175d0fbcd3756d5b4d7e1"}
Sep 18 14:03:10.621167 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:10.621144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5rdfc" event={"ID":"a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a","Type":"ContainerStarted","Data":"b0ff8897f88094d3956a22377e540d2ed8e5c30933ef537f65fde4094a91e1ea"}
Sep 18 14:03:13.640406 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:13.640318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg" event={"ID":"039e20df-12e8-4f1f-a0aa-a4e79c1c759f","Type":"ContainerStarted","Data":"de9596d1ac1619fc0334a4a946d12a16f20cd0de160abc5b301ed10f2c6c8685"}
Sep 18 14:03:13.641934 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:13.641910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5rdfc" event={"ID":"a6dc8439-e78a-4e7c-9ac0-d3e1254ea78a","Type":"ContainerStarted","Data":"e0f1e544089a0dd3698bfa2f28db19e5f3adcb59a9bd25ad73913bf706e38d98"}
Sep 18 14:03:13.656301 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:13.656258    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qzhxg" podStartSLOduration=2.8801396219999997 podStartE2EDuration="4.656246398s" totalImagesPullingTime="1.776106776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:13.6544927 +0000 UTC m=+281.811012981" watchObservedRunningTime="2026-09-18 14:03:13.656246398 +0000 UTC m=+281.812766679"
Sep 18 14:03:13.671067 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:13.671022    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-5rdfc" podStartSLOduration=1.34674579 podStartE2EDuration="4.671009713s" totalImagesPullingTime="3.324263923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:13.669286367 +0000 UTC m=+281.825806649" watchObservedRunningTime="2026-09-18 14:03:13.671009713 +0000 UTC m=+281.827529995"
Sep 18 14:03:15.688695 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.688654    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"]
Sep 18 14:03:15.707244 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.707213    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"]
Sep 18 14:03:15.707409 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.707346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:15.709836 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.709813    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 14:03:15.710229 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.710213    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 14:03:15.710747 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.710731    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4tjp\""
Sep 18 14:03:15.822066 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.822028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:15.822219 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.822104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:15.822219 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.822170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnnq6\" (UniqueName: \"kubernetes.io/projected/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-kube-api-access-mnnq6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:15.923488 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.923445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:15.923654 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.923500    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnnq6\" (UniqueName: \"kubernetes.io/projected/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-kube-api-access-mnnq6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:15.923654 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.923540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:15.923919 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.923897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:15.923954 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.923909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:15.932046 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:15.932022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnnq6\" (UniqueName: \"kubernetes.io/projected/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-kube-api-access-mnnq6\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:16.017027 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:16.016948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:16.307804 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:16.307760    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"]
Sep 18 14:03:16.310260 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:03:16.310227    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71506131_5f06_4cb0_9ab1_f5400eaf9dcd.slice/crio-235994a0f1ff81449df07aee501075f9aed2c3ac3b1f532f08bc846059fcf108 WatchSource:0}: Error finding container 235994a0f1ff81449df07aee501075f9aed2c3ac3b1f532f08bc846059fcf108: Status 404 returned error can't find the container with id 235994a0f1ff81449df07aee501075f9aed2c3ac3b1f532f08bc846059fcf108
Sep 18 14:03:16.656645 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:16.656562    2563 generic.go:356] "Generic (PLEG): container finished" podID="71506131-5f06-4cb0-9ab1-f5400eaf9dcd" containerID="58392b594767e8469d769bff06936c54df7e4376a3d497e682cd1ed6bdd6f4c6" exitCode=0
Sep 18 14:03:16.656645 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:16.656614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh" event={"ID":"71506131-5f06-4cb0-9ab1-f5400eaf9dcd","Type":"ContainerDied","Data":"58392b594767e8469d769bff06936c54df7e4376a3d497e682cd1ed6bdd6f4c6"}
Sep 18 14:03:16.656998 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:16.656658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh" event={"ID":"71506131-5f06-4cb0-9ab1-f5400eaf9dcd","Type":"ContainerStarted","Data":"235994a0f1ff81449df07aee501075f9aed2c3ac3b1f532f08bc846059fcf108"}
Sep 18 14:03:17.662693 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:17.662659    2563 generic.go:356] "Generic (PLEG): container finished" podID="71506131-5f06-4cb0-9ab1-f5400eaf9dcd" containerID="dc289d0f02f5fe3cc0a3eba471b8f20b61a5c986cc9651dfabf2669c468605e2" exitCode=0
Sep 18 14:03:17.663032 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:17.662724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh" event={"ID":"71506131-5f06-4cb0-9ab1-f5400eaf9dcd","Type":"ContainerDied","Data":"dc289d0f02f5fe3cc0a3eba471b8f20b61a5c986cc9651dfabf2669c468605e2"}
Sep 18 14:03:18.669383 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:18.669341    2563 generic.go:356] "Generic (PLEG): container finished" podID="71506131-5f06-4cb0-9ab1-f5400eaf9dcd" containerID="9450fc08cbdf685a05173c21fe254d4a153446d57e7c9348f206fff37b068d44" exitCode=0
Sep 18 14:03:18.669751 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:18.669427    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh" event={"ID":"71506131-5f06-4cb0-9ab1-f5400eaf9dcd","Type":"ContainerDied","Data":"9450fc08cbdf685a05173c21fe254d4a153446d57e7c9348f206fff37b068d44"}
Sep 18 14:03:19.797985 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:19.797960    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:19.953470 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:19.953403    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-bundle\") pod \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") "
Sep 18 14:03:19.953470 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:19.953466    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-util\") pod \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") "
Sep 18 14:03:19.953660 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:19.953484    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnnq6\" (UniqueName: \"kubernetes.io/projected/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-kube-api-access-mnnq6\") pod \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\" (UID: \"71506131-5f06-4cb0-9ab1-f5400eaf9dcd\") "
Sep 18 14:03:19.954156 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:19.954107    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-bundle" (OuterVolumeSpecName: "bundle") pod "71506131-5f06-4cb0-9ab1-f5400eaf9dcd" (UID: "71506131-5f06-4cb0-9ab1-f5400eaf9dcd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:03:19.955635 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:19.955606    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-kube-api-access-mnnq6" (OuterVolumeSpecName: "kube-api-access-mnnq6") pod "71506131-5f06-4cb0-9ab1-f5400eaf9dcd" (UID: "71506131-5f06-4cb0-9ab1-f5400eaf9dcd"). InnerVolumeSpecName "kube-api-access-mnnq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:03:19.959214 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:19.959186    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-util" (OuterVolumeSpecName: "util") pod "71506131-5f06-4cb0-9ab1-f5400eaf9dcd" (UID: "71506131-5f06-4cb0-9ab1-f5400eaf9dcd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:03:20.054481 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:20.054458    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:03:20.054481 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:20.054478    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-util\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:03:20.054608 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:20.054486    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnnq6\" (UniqueName: \"kubernetes.io/projected/71506131-5f06-4cb0-9ab1-f5400eaf9dcd-kube-api-access-mnnq6\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:03:20.682035 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:20.681992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh" event={"ID":"71506131-5f06-4cb0-9ab1-f5400eaf9dcd","Type":"ContainerDied","Data":"235994a0f1ff81449df07aee501075f9aed2c3ac3b1f532f08bc846059fcf108"}
Sep 18 14:03:20.682035 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:20.682036    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflp5lh"
Sep 18 14:03:20.682245 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:20.682079    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="235994a0f1ff81449df07aee501075f9aed2c3ac3b1f532f08bc846059fcf108"
Sep 18 14:03:27.298304 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.298266    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"]
Sep 18 14:03:27.298681 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.298584    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71506131-5f06-4cb0-9ab1-f5400eaf9dcd" containerName="pull"
Sep 18 14:03:27.298681 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.298595    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71506131-5f06-4cb0-9ab1-f5400eaf9dcd" containerName="pull"
Sep 18 14:03:27.298681 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.298613    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71506131-5f06-4cb0-9ab1-f5400eaf9dcd" containerName="extract"
Sep 18 14:03:27.298681 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.298619    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71506131-5f06-4cb0-9ab1-f5400eaf9dcd" containerName="extract"
Sep 18 14:03:27.298681 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.298629    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71506131-5f06-4cb0-9ab1-f5400eaf9dcd" containerName="util"
Sep 18 14:03:27.298681 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.298635    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71506131-5f06-4cb0-9ab1-f5400eaf9dcd" containerName="util"
Sep 18 14:03:27.298681 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.298685    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="71506131-5f06-4cb0-9ab1-f5400eaf9dcd" containerName="extract"
Sep 18 14:03:27.301736 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.301719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:27.304482 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.304451    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 14:03:27.304619 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.304453    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 14:03:27.306005 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.305986    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4tjp\""
Sep 18 14:03:27.311214 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.311190    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"]
Sep 18 14:03:27.409483 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.409443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:27.409658 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.409499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wvz\" (UniqueName: \"kubernetes.io/projected/22d33674-2fbe-4454-8442-df5f5ed2dbc5-kube-api-access-t7wvz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:27.409658 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.409526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:27.509980 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.509935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wvz\" (UniqueName: \"kubernetes.io/projected/22d33674-2fbe-4454-8442-df5f5ed2dbc5-kube-api-access-t7wvz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:27.510170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.509991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:27.510170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.510054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:27.510450 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.510430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:27.510484 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.510471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:27.521872 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.521848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wvz\" (UniqueName: \"kubernetes.io/projected/22d33674-2fbe-4454-8442-df5f5ed2dbc5-kube-api-access-t7wvz\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:27.611650 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.611563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:27.882163 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:27.882119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"]
Sep 18 14:03:27.884404 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:03:27.884373    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d33674_2fbe_4454_8442_df5f5ed2dbc5.slice/crio-a727989a0f287f17c88baf48f27485ef1bc7187f1d8908297be8b5dd50ba327d WatchSource:0}: Error finding container a727989a0f287f17c88baf48f27485ef1bc7187f1d8908297be8b5dd50ba327d: Status 404 returned error can't find the container with id a727989a0f287f17c88baf48f27485ef1bc7187f1d8908297be8b5dd50ba327d
Sep 18 14:03:28.722680 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:28.722648    2563 generic.go:356] "Generic (PLEG): container finished" podID="22d33674-2fbe-4454-8442-df5f5ed2dbc5" containerID="22e8e78ad2f54026efd63fa56cbce58a2d3f95f64bb74f8c20db8369516c7c9a" exitCode=0
Sep 18 14:03:28.723087 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:28.722715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn" event={"ID":"22d33674-2fbe-4454-8442-df5f5ed2dbc5","Type":"ContainerDied","Data":"22e8e78ad2f54026efd63fa56cbce58a2d3f95f64bb74f8c20db8369516c7c9a"}
Sep 18 14:03:28.723087 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:28.722762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn" event={"ID":"22d33674-2fbe-4454-8442-df5f5ed2dbc5","Type":"ContainerStarted","Data":"a727989a0f287f17c88baf48f27485ef1bc7187f1d8908297be8b5dd50ba327d"}
Sep 18 14:03:29.132331 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.132298    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"]
Sep 18 14:03:29.134332 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.134315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.140555 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.140494    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 14:03:29.141028 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.141007    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 14:03:29.141154 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.141028    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 14:03:29.141295 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.141274    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ms7h5\""
Sep 18 14:03:29.141397 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.141311    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 14:03:29.154948 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.154925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"]
Sep 18 14:03:29.222782 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.222753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcbfc919-f258-4ace-ba7c-bce32f56f64d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.222887 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.222798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcbfc919-f258-4ace-ba7c-bce32f56f64d-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.222887 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.222854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcbfc919-f258-4ace-ba7c-bce32f56f64d-tmp\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.223000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.222894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfzgq\" (UniqueName: \"kubernetes.io/projected/bcbfc919-f258-4ace-ba7c-bce32f56f64d-kube-api-access-pfzgq\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.223000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.222956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcbfc919-f258-4ace-ba7c-bce32f56f64d-manifests\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.323953 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.323930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcbfc919-f258-4ace-ba7c-bce32f56f64d-manifests\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.324047 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.323972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcbfc919-f258-4ace-ba7c-bce32f56f64d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.324047 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.323995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcbfc919-f258-4ace-ba7c-bce32f56f64d-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.324047 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.324018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcbfc919-f258-4ace-ba7c-bce32f56f64d-tmp\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.324047 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.324043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfzgq\" (UniqueName: \"kubernetes.io/projected/bcbfc919-f258-4ace-ba7c-bce32f56f64d-kube-api-access-pfzgq\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.324330 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.324309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcbfc919-f258-4ace-ba7c-bce32f56f64d-manifests\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.326218 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.326192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcbfc919-f258-4ace-ba7c-bce32f56f64d-tmp\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.326218 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.326205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcbfc919-f258-4ace-ba7c-bce32f56f64d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.326335 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.326307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcbfc919-f258-4ace-ba7c-bce32f56f64d-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.337221 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.337197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfzgq\" (UniqueName: \"kubernetes.io/projected/bcbfc919-f258-4ace-ba7c-bce32f56f64d-kube-api-access-pfzgq\") pod \"opendatahub-operator-controller-manager-74d96fdcf9-x4ffv\" (UID: \"bcbfc919-f258-4ace-ba7c-bce32f56f64d\") " pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.444091 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.444059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:29.717436 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.717412    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"]
Sep 18 14:03:29.719304 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:03:29.719278    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbfc919_f258_4ace_ba7c_bce32f56f64d.slice/crio-0cd91de914b10a613eea0a458156a723fd8bca200f97bc1517ebf3d0994a4b87 WatchSource:0}: Error finding container 0cd91de914b10a613eea0a458156a723fd8bca200f97bc1517ebf3d0994a4b87: Status 404 returned error can't find the container with id 0cd91de914b10a613eea0a458156a723fd8bca200f97bc1517ebf3d0994a4b87
Sep 18 14:03:29.729070 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.729041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv" event={"ID":"bcbfc919-f258-4ace-ba7c-bce32f56f64d","Type":"ContainerStarted","Data":"0cd91de914b10a613eea0a458156a723fd8bca200f97bc1517ebf3d0994a4b87"}
Sep 18 14:03:29.731085 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.731061    2563 generic.go:356] "Generic (PLEG): container finished" podID="22d33674-2fbe-4454-8442-df5f5ed2dbc5" containerID="dfbc575db7c273ff9d8122cf2bcebbcd510fab5092f2f8943cd95121c396e4d9" exitCode=0
Sep 18 14:03:29.731192 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:29.731167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn" event={"ID":"22d33674-2fbe-4454-8442-df5f5ed2dbc5","Type":"ContainerDied","Data":"dfbc575db7c273ff9d8122cf2bcebbcd510fab5092f2f8943cd95121c396e4d9"}
Sep 18 14:03:30.740515 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:30.740477    2563 generic.go:356] "Generic (PLEG): container finished" podID="22d33674-2fbe-4454-8442-df5f5ed2dbc5" containerID="8969051b8721faf089104bfc6277f6146071c6907683f22613a3babe4bc61124" exitCode=0
Sep 18 14:03:30.740968 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:30.740547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn" event={"ID":"22d33674-2fbe-4454-8442-df5f5ed2dbc5","Type":"ContainerDied","Data":"8969051b8721faf089104bfc6277f6146071c6907683f22613a3babe4bc61124"}
Sep 18 14:03:32.572963 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.572938    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:03:32.573393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.573211    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:03:32.573393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.573264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:32.587537 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.587515    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 14:03:32.648040 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.648019    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-bundle\") pod \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") "
Sep 18 14:03:32.648170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.648063    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-util\") pod \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") "
Sep 18 14:03:32.648170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.648095    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7wvz\" (UniqueName: \"kubernetes.io/projected/22d33674-2fbe-4454-8442-df5f5ed2dbc5-kube-api-access-t7wvz\") pod \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\" (UID: \"22d33674-2fbe-4454-8442-df5f5ed2dbc5\") "
Sep 18 14:03:32.648895 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.648870    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-bundle" (OuterVolumeSpecName: "bundle") pod "22d33674-2fbe-4454-8442-df5f5ed2dbc5" (UID: "22d33674-2fbe-4454-8442-df5f5ed2dbc5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:03:32.650335 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.650308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d33674-2fbe-4454-8442-df5f5ed2dbc5-kube-api-access-t7wvz" (OuterVolumeSpecName: "kube-api-access-t7wvz") pod "22d33674-2fbe-4454-8442-df5f5ed2dbc5" (UID: "22d33674-2fbe-4454-8442-df5f5ed2dbc5"). InnerVolumeSpecName "kube-api-access-t7wvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:03:32.654137 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.654106    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-util" (OuterVolumeSpecName: "util") pod "22d33674-2fbe-4454-8442-df5f5ed2dbc5" (UID: "22d33674-2fbe-4454-8442-df5f5ed2dbc5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:03:32.749341 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.749307    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:03:32.749341 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.749339    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22d33674-2fbe-4454-8442-df5f5ed2dbc5-util\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:03:32.749341 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.749351    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wvz\" (UniqueName: \"kubernetes.io/projected/22d33674-2fbe-4454-8442-df5f5ed2dbc5-kube-api-access-t7wvz\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:03:32.751736 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.751704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv" event={"ID":"bcbfc919-f258-4ace-ba7c-bce32f56f64d","Type":"ContainerStarted","Data":"de325302f3d86c60e0936a09212d04768cbcdfa46e0f620075c9140122034ed5"}
Sep 18 14:03:32.753558 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.753532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn" event={"ID":"22d33674-2fbe-4454-8442-df5f5ed2dbc5","Type":"ContainerDied","Data":"a727989a0f287f17c88baf48f27485ef1bc7187f1d8908297be8b5dd50ba327d"}
Sep 18 14:03:32.753558 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.753554    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ljqrn"
Sep 18 14:03:32.753663 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:32.753578    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a727989a0f287f17c88baf48f27485ef1bc7187f1d8908297be8b5dd50ba327d"
Sep 18 14:03:33.765555 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:33.765525    2563 generic.go:356] "Generic (PLEG): container finished" podID="bcbfc919-f258-4ace-ba7c-bce32f56f64d" containerID="de325302f3d86c60e0936a09212d04768cbcdfa46e0f620075c9140122034ed5" exitCode=0
Sep 18 14:03:33.765970 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:33.765569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv" event={"ID":"bcbfc919-f258-4ace-ba7c-bce32f56f64d","Type":"ContainerDied","Data":"de325302f3d86c60e0936a09212d04768cbcdfa46e0f620075c9140122034ed5"}
Sep 18 14:03:33.766189 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:33.766169    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:03:34.772854 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:34.772816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv" event={"ID":"bcbfc919-f258-4ace-ba7c-bce32f56f64d","Type":"ContainerStarted","Data":"cd2719b5b8953a66dddff28c424e88dd64e9a6cfb9aa2f932ddc7cf233086a36"}
Sep 18 14:03:34.773304 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:34.772894    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:34.798970 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:34.798929    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv" podStartSLOduration=1.6086647059999999 podStartE2EDuration="5.798916288s" totalImagesPullingTime="3.159873883s" totalInitContainerRuntime="1.030377699s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:34.797906586 +0000 UTC m=+302.954426879" watchObservedRunningTime="2026-09-18 14:03:34.798916288 +0000 UTC m=+302.955436569"
Sep 18 14:03:45.780026 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:45.779995    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74d96fdcf9-x4ffv"
Sep 18 14:03:48.364202 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.364164    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"]
Sep 18 14:03:48.364631 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.364599    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d33674-2fbe-4454-8442-df5f5ed2dbc5" containerName="util"
Sep 18 14:03:48.364631 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.364614    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d33674-2fbe-4454-8442-df5f5ed2dbc5" containerName="util"
Sep 18 14:03:48.364732 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.364633    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d33674-2fbe-4454-8442-df5f5ed2dbc5" containerName="pull"
Sep 18 14:03:48.364732 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.364642    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d33674-2fbe-4454-8442-df5f5ed2dbc5" containerName="pull"
Sep 18 14:03:48.364732 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.364656    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d33674-2fbe-4454-8442-df5f5ed2dbc5" containerName="extract"
Sep 18 14:03:48.364732 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.364664    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d33674-2fbe-4454-8442-df5f5ed2dbc5" containerName="extract"
Sep 18 14:03:48.364865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.364746    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d33674-2fbe-4454-8442-df5f5ed2dbc5" containerName="extract"
Sep 18 14:03:48.369862 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.369837    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:48.372587 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.372564    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 14:03:48.376966 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.376942    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 14:03:48.377292 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.377267    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h4m78\""
Sep 18 14:03:48.389327 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.389305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"]
Sep 18 14:03:48.474911 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.474879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586bc67f-67ae-4daf-a65d-4791833eae78-tmp\") pod \"kube-auth-proxy-86dc75d4b5-bzxf7\" (UID: \"586bc67f-67ae-4daf-a65d-4791833eae78\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:48.475041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.474916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvx47\" (UniqueName: \"kubernetes.io/projected/586bc67f-67ae-4daf-a65d-4791833eae78-kube-api-access-vvx47\") pod \"kube-auth-proxy-86dc75d4b5-bzxf7\" (UID: \"586bc67f-67ae-4daf-a65d-4791833eae78\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:48.475041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.474947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586bc67f-67ae-4daf-a65d-4791833eae78-tls-certs\") pod \"kube-auth-proxy-86dc75d4b5-bzxf7\" (UID: \"586bc67f-67ae-4daf-a65d-4791833eae78\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:48.576226 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.576193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586bc67f-67ae-4daf-a65d-4791833eae78-tmp\") pod \"kube-auth-proxy-86dc75d4b5-bzxf7\" (UID: \"586bc67f-67ae-4daf-a65d-4791833eae78\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:48.576378 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.576229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvx47\" (UniqueName: \"kubernetes.io/projected/586bc67f-67ae-4daf-a65d-4791833eae78-kube-api-access-vvx47\") pod \"kube-auth-proxy-86dc75d4b5-bzxf7\" (UID: \"586bc67f-67ae-4daf-a65d-4791833eae78\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:48.576378 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.576351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586bc67f-67ae-4daf-a65d-4791833eae78-tls-certs\") pod \"kube-auth-proxy-86dc75d4b5-bzxf7\" (UID: \"586bc67f-67ae-4daf-a65d-4791833eae78\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:48.576494 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:03:48.576459    2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 18 14:03:48.576545 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:03:48.576520    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/586bc67f-67ae-4daf-a65d-4791833eae78-tls-certs podName:586bc67f-67ae-4daf-a65d-4791833eae78 nodeName:}" failed. No retries permitted until 2026-09-18 14:03:49.076504455 +0000 UTC m=+317.233024714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/586bc67f-67ae-4daf-a65d-4791833eae78-tls-certs") pod "kube-auth-proxy-86dc75d4b5-bzxf7" (UID: "586bc67f-67ae-4daf-a65d-4791833eae78") : secret "kube-auth-proxy-tls" not found
Sep 18 14:03:48.578399 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.578381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/586bc67f-67ae-4daf-a65d-4791833eae78-tmp\") pod \"kube-auth-proxy-86dc75d4b5-bzxf7\" (UID: \"586bc67f-67ae-4daf-a65d-4791833eae78\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:48.594424 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:48.594396    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvx47\" (UniqueName: \"kubernetes.io/projected/586bc67f-67ae-4daf-a65d-4791833eae78-kube-api-access-vvx47\") pod \"kube-auth-proxy-86dc75d4b5-bzxf7\" (UID: \"586bc67f-67ae-4daf-a65d-4791833eae78\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:49.080593 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:49.080560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586bc67f-67ae-4daf-a65d-4791833eae78-tls-certs\") pod \"kube-auth-proxy-86dc75d4b5-bzxf7\" (UID: \"586bc67f-67ae-4daf-a65d-4791833eae78\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:49.083077 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:49.083047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/586bc67f-67ae-4daf-a65d-4791833eae78-tls-certs\") pod \"kube-auth-proxy-86dc75d4b5-bzxf7\" (UID: \"586bc67f-67ae-4daf-a65d-4791833eae78\") " pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:49.281874 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:49.281846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"
Sep 18 14:03:49.566261 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:49.566228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7"]
Sep 18 14:03:49.569276 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:03:49.569243    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586bc67f_67ae_4daf_a65d_4791833eae78.slice/crio-904dadb23725ddb9d65e21dbf10612e0fb5691c094da14e2c416630863993652 WatchSource:0}: Error finding container 904dadb23725ddb9d65e21dbf10612e0fb5691c094da14e2c416630863993652: Status 404 returned error can't find the container with id 904dadb23725ddb9d65e21dbf10612e0fb5691c094da14e2c416630863993652
Sep 18 14:03:49.843862 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:49.843828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7" event={"ID":"586bc67f-67ae-4daf-a65d-4791833eae78","Type":"ContainerStarted","Data":"904dadb23725ddb9d65e21dbf10612e0fb5691c094da14e2c416630863993652"}
Sep 18 14:03:50.800781 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:50.800748    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"]
Sep 18 14:03:50.803940 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:50.803915    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:50.806970 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:50.806922    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mmxrk\""
Sep 18 14:03:50.807792 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:50.807649    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 14:03:50.807792 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:50.807672    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 14:03:50.807792 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:50.807673    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 14:03:50.813466 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:50.813395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"]
Sep 18 14:03:50.997474 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:50.997437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3af5117d-baa2-4ed3-872b-896519bbc857-metrics-cert\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:50.997649 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:50.997502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4sw\" (UniqueName: \"kubernetes.io/projected/3af5117d-baa2-4ed3-872b-896519bbc857-kube-api-access-rk4sw\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:50.997649 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:50.997550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3af5117d-baa2-4ed3-872b-896519bbc857-manager-config\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:50.997649 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:50.997621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3af5117d-baa2-4ed3-872b-896519bbc857-cert\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:51.098637 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:51.098554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3af5117d-baa2-4ed3-872b-896519bbc857-manager-config\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:51.098637 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:51.098621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3af5117d-baa2-4ed3-872b-896519bbc857-cert\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:51.098856 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:51.098686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3af5117d-baa2-4ed3-872b-896519bbc857-metrics-cert\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:51.098856 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:51.098718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4sw\" (UniqueName: \"kubernetes.io/projected/3af5117d-baa2-4ed3-872b-896519bbc857-kube-api-access-rk4sw\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:51.099417 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:51.099389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3af5117d-baa2-4ed3-872b-896519bbc857-manager-config\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:51.102259 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:51.101994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3af5117d-baa2-4ed3-872b-896519bbc857-metrics-cert\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:51.103266 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:51.103219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3af5117d-baa2-4ed3-872b-896519bbc857-cert\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:51.109299 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:51.109274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4sw\" (UniqueName: \"kubernetes.io/projected/3af5117d-baa2-4ed3-872b-896519bbc857-kube-api-access-rk4sw\") pod \"lws-controller-manager-ff9cccccb-624x7\" (UID: \"3af5117d-baa2-4ed3-872b-896519bbc857\") " pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:51.118638 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:51.118613    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:51.907283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:51.907253    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"]
Sep 18 14:03:52.376659 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:03:52.376627    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af5117d_baa2_4ed3_872b_896519bbc857.slice/crio-fb47e1a73e60f0a55be750b7739ae81a536b3e7c1cab5266ba9f17cb9a5f9896 WatchSource:0}: Error finding container fb47e1a73e60f0a55be750b7739ae81a536b3e7c1cab5266ba9f17cb9a5f9896: Status 404 returned error can't find the container with id fb47e1a73e60f0a55be750b7739ae81a536b3e7c1cab5266ba9f17cb9a5f9896
Sep 18 14:03:52.863535 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:52.863497    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7" event={"ID":"586bc67f-67ae-4daf-a65d-4791833eae78","Type":"ContainerStarted","Data":"f3ad20aeb13ecf424b1b9b2e0f5fe7b8471753a52fdc7b2c3b866322e39348cb"}
Sep 18 14:03:52.864761 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:52.864737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7" event={"ID":"3af5117d-baa2-4ed3-872b-896519bbc857","Type":"ContainerStarted","Data":"fb47e1a73e60f0a55be750b7739ae81a536b3e7c1cab5266ba9f17cb9a5f9896"}
Sep 18 14:03:52.880564 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:52.880515    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86dc75d4b5-bzxf7" podStartSLOduration=2.022618125 podStartE2EDuration="4.880501662s" totalImagesPullingTime="2.857883537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:52.87966667 +0000 UTC m=+321.036186952" watchObservedRunningTime="2026-09-18 14:03:52.880501662 +0000 UTC m=+321.037021988"
Sep 18 14:03:53.872038 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:53.872003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7" event={"ID":"3af5117d-baa2-4ed3-872b-896519bbc857","Type":"ContainerStarted","Data":"d3f9b8b970da827960d0552bde9295551c5900657a206d89bf8b16dec02fc37a"}
Sep 18 14:03:53.872493 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:53.872311    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:03:53.890657 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:03:53.890608    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7" podStartSLOduration=2.518207265 podStartE2EDuration="3.890592579s" totalImagesPullingTime="1.372385314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:03:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:03:53.889687908 +0000 UTC m=+322.046208190" watchObservedRunningTime="2026-09-18 14:03:53.890592579 +0000 UTC m=+322.047112860"
Sep 18 14:04:04.879889 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:04:04.879850    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-ff9cccccb-624x7"
Sep 18 14:05:04.008633 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.008596    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"]
Sep 18 14:05:04.011961 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.011943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:04.014464 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.014444    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 14:05:04.014567 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.014516    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 14:05:04.015310 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.015296    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4tjp\""
Sep 18 14:05:04.019577 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.019554    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"]
Sep 18 14:05:04.108846 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.108811    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"]
Sep 18 14:05:04.112140 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.112108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:04.120558 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.120531    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"]
Sep 18 14:05:04.175112 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.175082    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:04.175269 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.175221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:04.175269 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.175257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27wcb\" (UniqueName: \"kubernetes.io/projected/bfaf315b-d1e8-45b6-b99b-14d16e01a624-kube-api-access-27wcb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:04.275943 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.275855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:04.275943 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.275900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9qbm\" (UniqueName: \"kubernetes.io/projected/5494c940-4ead-447c-a1a3-e2c2193ae8df-kube-api-access-f9qbm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:04.276185 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.275976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:04.276185 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.276033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:04.276185 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.276066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:04.276185 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.276088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27wcb\" (UniqueName: \"kubernetes.io/projected/bfaf315b-d1e8-45b6-b99b-14d16e01a624-kube-api-access-27wcb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:04.276383 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.276240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:04.276383 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.276360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:04.285900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.285867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27wcb\" (UniqueName: \"kubernetes.io/projected/bfaf315b-d1e8-45b6-b99b-14d16e01a624-kube-api-access-27wcb\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:04.302144 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.302102    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"]
Sep 18 14:05:04.305480 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.305462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:04.315854 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.315832    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"]
Sep 18 14:05:04.321595 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.321575    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:04.369336 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.369292    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"]
Sep 18 14:05:04.373532 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.373503    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:04.377307 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.377268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:04.377425 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.377314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:04.377425 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.377385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9qbm\" (UniqueName: \"kubernetes.io/projected/5494c940-4ead-447c-a1a3-e2c2193ae8df-kube-api-access-f9qbm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:04.378087 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.378058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:04.378222 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.378064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:04.382270 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.382243    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"]
Sep 18 14:05:04.389812 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.389781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9qbm\" (UniqueName: \"kubernetes.io/projected/5494c940-4ead-447c-a1a3-e2c2193ae8df-kube-api-access-f9qbm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:04.422058 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.422021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:04.479059 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.478589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:04.479059 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.478648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:04.479059 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.478682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:04.479059 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.478737    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j47j2\" (UniqueName: \"kubernetes.io/projected/1701344e-60a8-496a-be8c-bd4d9db4eee2-kube-api-access-j47j2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:04.479059 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.478787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhgrq\" (UniqueName: \"kubernetes.io/projected/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-kube-api-access-qhgrq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:04.479059 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.478830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:04.580164 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.580078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:04.580164 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.580118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:04.580164 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.580157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:04.580419 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.580187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47j2\" (UniqueName: \"kubernetes.io/projected/1701344e-60a8-496a-be8c-bd4d9db4eee2-kube-api-access-j47j2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:04.580419 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.580216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhgrq\" (UniqueName: \"kubernetes.io/projected/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-kube-api-access-qhgrq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:04.580419 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.580242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:04.580575 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.580528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:04.580575 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.580535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:04.580666 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.580589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:04.580666 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.580596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:04.590527 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.590497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhgrq\" (UniqueName: \"kubernetes.io/projected/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-kube-api-access-qhgrq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:04.591112 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.591096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47j2\" (UniqueName: \"kubernetes.io/projected/1701344e-60a8-496a-be8c-bd4d9db4eee2-kube-api-access-j47j2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:04.603228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.603202    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"]
Sep 18 14:05:04.605035 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:05:04.605009    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaf315b_d1e8_45b6_b99b_14d16e01a624.slice/crio-57648321b59dff7fdc145a540feff08c7e89174f9054452a8619494b67fbd696 WatchSource:0}: Error finding container 57648321b59dff7fdc145a540feff08c7e89174f9054452a8619494b67fbd696: Status 404 returned error can't find the container with id 57648321b59dff7fdc145a540feff08c7e89174f9054452a8619494b67fbd696
Sep 18 14:05:04.614393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.614373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:04.690576 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.690553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:04.700027 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.700004    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"]
Sep 18 14:05:04.701979 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:05:04.701953    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5494c940_4ead_447c_a1a3_e2c2193ae8df.slice/crio-7a5d8e382c254ca9895905f5ef8e934b65844c4c79d879e75f63a84fade72a9d WatchSource:0}: Error finding container 7a5d8e382c254ca9895905f5ef8e934b65844c4c79d879e75f63a84fade72a9d: Status 404 returned error can't find the container with id 7a5d8e382c254ca9895905f5ef8e934b65844c4c79d879e75f63a84fade72a9d
Sep 18 14:05:04.901849 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:04.901823    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"]
Sep 18 14:05:04.923893 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:05:04.923856    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1701344e_60a8_496a_be8c_bd4d9db4eee2.slice/crio-ccd2fed98fbedb3619678e11a96f58496633dd58419734cdb00a395671efdac9 WatchSource:0}: Error finding container ccd2fed98fbedb3619678e11a96f58496633dd58419734cdb00a395671efdac9: Status 404 returned error can't find the container with id ccd2fed98fbedb3619678e11a96f58496633dd58419734cdb00a395671efdac9
Sep 18 14:05:05.013792 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.013738    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"]
Sep 18 14:05:05.016057 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:05:05.016019    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6841c85b_6f9d_41a7_a2fe_cf9ddf0036fb.slice/crio-7649a4188f6e5f85336cd83738f8780a14997266bac5f675eb7078a3e67a315d WatchSource:0}: Error finding container 7649a4188f6e5f85336cd83738f8780a14997266bac5f675eb7078a3e67a315d: Status 404 returned error can't find the container with id 7649a4188f6e5f85336cd83738f8780a14997266bac5f675eb7078a3e67a315d
Sep 18 14:05:05.219001 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.218965    2563 generic.go:356] "Generic (PLEG): container finished" podID="6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" containerID="45770edf857c3517b99878128809fbae3b6551a95928df4714f1c8adc99ccb8d" exitCode=0
Sep 18 14:05:05.219170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.219050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs" event={"ID":"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb","Type":"ContainerDied","Data":"45770edf857c3517b99878128809fbae3b6551a95928df4714f1c8adc99ccb8d"}
Sep 18 14:05:05.219170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.219099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs" event={"ID":"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb","Type":"ContainerStarted","Data":"7649a4188f6e5f85336cd83738f8780a14997266bac5f675eb7078a3e67a315d"}
Sep 18 14:05:05.220703 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.220681    2563 generic.go:356] "Generic (PLEG): container finished" podID="1701344e-60a8-496a-be8c-bd4d9db4eee2" containerID="d6a5f9b4cbd4aeb49a8a2f9f89935488561a7d37078b4cc76d9944bb6eb6a4f4" exitCode=0
Sep 18 14:05:05.220805 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.220771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8" event={"ID":"1701344e-60a8-496a-be8c-bd4d9db4eee2","Type":"ContainerDied","Data":"d6a5f9b4cbd4aeb49a8a2f9f89935488561a7d37078b4cc76d9944bb6eb6a4f4"}
Sep 18 14:05:05.220865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.220815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8" event={"ID":"1701344e-60a8-496a-be8c-bd4d9db4eee2","Type":"ContainerStarted","Data":"ccd2fed98fbedb3619678e11a96f58496633dd58419734cdb00a395671efdac9"}
Sep 18 14:05:05.222473 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.222447    2563 generic.go:356] "Generic (PLEG): container finished" podID="5494c940-4ead-447c-a1a3-e2c2193ae8df" containerID="42cc7b6b5ca4f7ff6a561ce7b620eee46a2c9b6005df40dcda634d54b4f0680f" exitCode=0
Sep 18 14:05:05.222569 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.222515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8" event={"ID":"5494c940-4ead-447c-a1a3-e2c2193ae8df","Type":"ContainerDied","Data":"42cc7b6b5ca4f7ff6a561ce7b620eee46a2c9b6005df40dcda634d54b4f0680f"}
Sep 18 14:05:05.222569 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.222556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8" event={"ID":"5494c940-4ead-447c-a1a3-e2c2193ae8df","Type":"ContainerStarted","Data":"7a5d8e382c254ca9895905f5ef8e934b65844c4c79d879e75f63a84fade72a9d"}
Sep 18 14:05:05.224159 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.224122    2563 generic.go:356] "Generic (PLEG): container finished" podID="bfaf315b-d1e8-45b6-b99b-14d16e01a624" containerID="44c44538aa83d79d2c2480b8cfaa44f1c6b5eb379653a4b79c0e67fd92efe3cf" exitCode=0
Sep 18 14:05:05.224249 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.224157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg" event={"ID":"bfaf315b-d1e8-45b6-b99b-14d16e01a624","Type":"ContainerDied","Data":"44c44538aa83d79d2c2480b8cfaa44f1c6b5eb379653a4b79c0e67fd92efe3cf"}
Sep 18 14:05:05.224249 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:05.224201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg" event={"ID":"bfaf315b-d1e8-45b6-b99b-14d16e01a624","Type":"ContainerStarted","Data":"57648321b59dff7fdc145a540feff08c7e89174f9054452a8619494b67fbd696"}
Sep 18 14:05:06.231437 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:06.231407    2563 generic.go:356] "Generic (PLEG): container finished" podID="1701344e-60a8-496a-be8c-bd4d9db4eee2" containerID="47980e6b4026707655c3e6536c701899817be77da75a66a270df989a7994d03c" exitCode=0
Sep 18 14:05:06.231809 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:06.231492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8" event={"ID":"1701344e-60a8-496a-be8c-bd4d9db4eee2","Type":"ContainerDied","Data":"47980e6b4026707655c3e6536c701899817be77da75a66a270df989a7994d03c"}
Sep 18 14:05:06.233662 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:06.233637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs" event={"ID":"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb","Type":"ContainerStarted","Data":"d41be6f05cb762b8919625e59e218d1c64d7b527bef1c6de0f54f69cbac1c030"}
Sep 18 14:05:07.243494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:07.243445    2563 generic.go:356] "Generic (PLEG): container finished" podID="bfaf315b-d1e8-45b6-b99b-14d16e01a624" containerID="ee2b87ef47fad5eb1e041c52402ee48cbea65abe9eba60a0528c74e9f0fcfd9b" exitCode=0
Sep 18 14:05:07.243931 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:07.243519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg" event={"ID":"bfaf315b-d1e8-45b6-b99b-14d16e01a624","Type":"ContainerDied","Data":"ee2b87ef47fad5eb1e041c52402ee48cbea65abe9eba60a0528c74e9f0fcfd9b"}
Sep 18 14:05:07.245577 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:07.245549    2563 generic.go:356] "Generic (PLEG): container finished" podID="6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" containerID="d41be6f05cb762b8919625e59e218d1c64d7b527bef1c6de0f54f69cbac1c030" exitCode=0
Sep 18 14:05:07.245690 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:07.245668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs" event={"ID":"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb","Type":"ContainerDied","Data":"d41be6f05cb762b8919625e59e218d1c64d7b527bef1c6de0f54f69cbac1c030"}
Sep 18 14:05:07.247999 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:07.247978    2563 generic.go:356] "Generic (PLEG): container finished" podID="1701344e-60a8-496a-be8c-bd4d9db4eee2" containerID="e26d009922734d91c7ca8ae8977c7e7563dd2348f211aaa2d67717b827d4f040" exitCode=0
Sep 18 14:05:07.248083 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:07.248016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8" event={"ID":"1701344e-60a8-496a-be8c-bd4d9db4eee2","Type":"ContainerDied","Data":"e26d009922734d91c7ca8ae8977c7e7563dd2348f211aaa2d67717b827d4f040"}
Sep 18 14:05:08.255093 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.255046    2563 generic.go:356] "Generic (PLEG): container finished" podID="bfaf315b-d1e8-45b6-b99b-14d16e01a624" containerID="c564cec3d612def9fe83d629393330ee5374ad89268633d4aa3683677183375b" exitCode=0
Sep 18 14:05:08.255564 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.255149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg" event={"ID":"bfaf315b-d1e8-45b6-b99b-14d16e01a624","Type":"ContainerDied","Data":"c564cec3d612def9fe83d629393330ee5374ad89268633d4aa3683677183375b"}
Sep 18 14:05:08.257372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.257349    2563 generic.go:356] "Generic (PLEG): container finished" podID="6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" containerID="3ef1d62b025ad2f6c9edca1144a57a3500ab0ead31e428cef54425bb549daed7" exitCode=0
Sep 18 14:05:08.257504 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.257438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs" event={"ID":"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb","Type":"ContainerDied","Data":"3ef1d62b025ad2f6c9edca1144a57a3500ab0ead31e428cef54425bb549daed7"}
Sep 18 14:05:08.389275 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.389247    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:08.518205 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.518081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-bundle\") pod \"1701344e-60a8-496a-be8c-bd4d9db4eee2\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") "
Sep 18 14:05:08.518205 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.518181    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j47j2\" (UniqueName: \"kubernetes.io/projected/1701344e-60a8-496a-be8c-bd4d9db4eee2-kube-api-access-j47j2\") pod \"1701344e-60a8-496a-be8c-bd4d9db4eee2\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") "
Sep 18 14:05:08.518389 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.518261    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-util\") pod \"1701344e-60a8-496a-be8c-bd4d9db4eee2\" (UID: \"1701344e-60a8-496a-be8c-bd4d9db4eee2\") "
Sep 18 14:05:08.518623 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.518589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-bundle" (OuterVolumeSpecName: "bundle") pod "1701344e-60a8-496a-be8c-bd4d9db4eee2" (UID: "1701344e-60a8-496a-be8c-bd4d9db4eee2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:08.520415 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.520384    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1701344e-60a8-496a-be8c-bd4d9db4eee2-kube-api-access-j47j2" (OuterVolumeSpecName: "kube-api-access-j47j2") pod "1701344e-60a8-496a-be8c-bd4d9db4eee2" (UID: "1701344e-60a8-496a-be8c-bd4d9db4eee2"). InnerVolumeSpecName "kube-api-access-j47j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:05:08.523653 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.523625    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-util" (OuterVolumeSpecName: "util") pod "1701344e-60a8-496a-be8c-bd4d9db4eee2" (UID: "1701344e-60a8-496a-be8c-bd4d9db4eee2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:08.619029 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.618993    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-util\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:08.619029 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.619023    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1701344e-60a8-496a-be8c-bd4d9db4eee2-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:08.619029 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:08.619031    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j47j2\" (UniqueName: \"kubernetes.io/projected/1701344e-60a8-496a-be8c-bd4d9db4eee2-kube-api-access-j47j2\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:09.264642 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.264608    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8"
Sep 18 14:05:09.264642 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.264636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62wcj8" event={"ID":"1701344e-60a8-496a-be8c-bd4d9db4eee2","Type":"ContainerDied","Data":"ccd2fed98fbedb3619678e11a96f58496633dd58419734cdb00a395671efdac9"}
Sep 18 14:05:09.265165 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.264710    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccd2fed98fbedb3619678e11a96f58496633dd58419734cdb00a395671efdac9"
Sep 18 14:05:09.418139 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.418107    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:09.420452 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.420434    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:09.526418 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.526324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-util\") pod \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") "
Sep 18 14:05:09.526418 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.526375    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-bundle\") pod \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") "
Sep 18 14:05:09.526418 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.526403    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-util\") pod \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") "
Sep 18 14:05:09.526693 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.526440    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhgrq\" (UniqueName: \"kubernetes.io/projected/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-kube-api-access-qhgrq\") pod \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") "
Sep 18 14:05:09.526693 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.526494    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27wcb\" (UniqueName: \"kubernetes.io/projected/bfaf315b-d1e8-45b6-b99b-14d16e01a624-kube-api-access-27wcb\") pod \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\" (UID: \"bfaf315b-d1e8-45b6-b99b-14d16e01a624\") "
Sep 18 14:05:09.526693 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.526560    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-bundle\") pod \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\" (UID: \"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb\") "
Sep 18 14:05:09.527124 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.527096    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-bundle" (OuterVolumeSpecName: "bundle") pod "6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" (UID: "6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:09.527292 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.527264    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-bundle" (OuterVolumeSpecName: "bundle") pod "bfaf315b-d1e8-45b6-b99b-14d16e01a624" (UID: "bfaf315b-d1e8-45b6-b99b-14d16e01a624"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:09.528788 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.528765    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-kube-api-access-qhgrq" (OuterVolumeSpecName: "kube-api-access-qhgrq") pod "6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" (UID: "6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb"). InnerVolumeSpecName "kube-api-access-qhgrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:05:09.528864 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.528799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaf315b-d1e8-45b6-b99b-14d16e01a624-kube-api-access-27wcb" (OuterVolumeSpecName: "kube-api-access-27wcb") pod "bfaf315b-d1e8-45b6-b99b-14d16e01a624" (UID: "bfaf315b-d1e8-45b6-b99b-14d16e01a624"). InnerVolumeSpecName "kube-api-access-27wcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:05:09.535174 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.535146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-util" (OuterVolumeSpecName: "util") pod "6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" (UID: "6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:09.535595 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.535577    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-util" (OuterVolumeSpecName: "util") pod "bfaf315b-d1e8-45b6-b99b-14d16e01a624" (UID: "bfaf315b-d1e8-45b6-b99b-14d16e01a624"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:09.627541 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.627498    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27wcb\" (UniqueName: \"kubernetes.io/projected/bfaf315b-d1e8-45b6-b99b-14d16e01a624-kube-api-access-27wcb\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:09.627541 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.627532    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:09.627541 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.627541    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-util\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:09.627541 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.627549    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:09.627541 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.627556    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfaf315b-d1e8-45b6-b99b-14d16e01a624-util\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:09.627824 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:09.627564    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhgrq\" (UniqueName: \"kubernetes.io/projected/6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb-kube-api-access-qhgrq\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:10.272161 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:10.272098    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg"
Sep 18 14:05:10.272594 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:10.272100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acmddg" event={"ID":"bfaf315b-d1e8-45b6-b99b-14d16e01a624","Type":"ContainerDied","Data":"57648321b59dff7fdc145a540feff08c7e89174f9054452a8619494b67fbd696"}
Sep 18 14:05:10.272594 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:10.272249    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57648321b59dff7fdc145a540feff08c7e89174f9054452a8619494b67fbd696"
Sep 18 14:05:10.274102 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:10.274084    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs"
Sep 18 14:05:10.274195 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:10.274099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7klmxs" event={"ID":"6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb","Type":"ContainerDied","Data":"7649a4188f6e5f85336cd83738f8780a14997266bac5f675eb7078a3e67a315d"}
Sep 18 14:05:10.274195 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:10.274147    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7649a4188f6e5f85336cd83738f8780a14997266bac5f675eb7078a3e67a315d"
Sep 18 14:05:13.291707 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:13.291674    2563 generic.go:356] "Generic (PLEG): container finished" podID="5494c940-4ead-447c-a1a3-e2c2193ae8df" containerID="8ac8a590b3b4770c02b549c5de82e6b4a77c5c1a2a472b86edcf283640724ad2" exitCode=0
Sep 18 14:05:13.292109 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:13.291770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8" event={"ID":"5494c940-4ead-447c-a1a3-e2c2193ae8df","Type":"ContainerDied","Data":"8ac8a590b3b4770c02b549c5de82e6b4a77c5c1a2a472b86edcf283640724ad2"}
Sep 18 14:05:14.299013 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:14.298977    2563 generic.go:356] "Generic (PLEG): container finished" podID="5494c940-4ead-447c-a1a3-e2c2193ae8df" containerID="ff8e71a2001e7e7656fa260fd1cdf12548e59a7afcb5c73a7b94bf73f2d96af2" exitCode=0
Sep 18 14:05:14.299483 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:14.299057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8" event={"ID":"5494c940-4ead-447c-a1a3-e2c2193ae8df","Type":"ContainerDied","Data":"ff8e71a2001e7e7656fa260fd1cdf12548e59a7afcb5c73a7b94bf73f2d96af2"}
Sep 18 14:05:15.431178 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:15.431149    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:15.469632 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:15.469598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-bundle\") pod \"5494c940-4ead-447c-a1a3-e2c2193ae8df\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") "
Sep 18 14:05:15.469632 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:15.469635    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9qbm\" (UniqueName: \"kubernetes.io/projected/5494c940-4ead-447c-a1a3-e2c2193ae8df-kube-api-access-f9qbm\") pod \"5494c940-4ead-447c-a1a3-e2c2193ae8df\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") "
Sep 18 14:05:15.469827 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:15.469679    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-util\") pod \"5494c940-4ead-447c-a1a3-e2c2193ae8df\" (UID: \"5494c940-4ead-447c-a1a3-e2c2193ae8df\") "
Sep 18 14:05:15.470293 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:15.470257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-bundle" (OuterVolumeSpecName: "bundle") pod "5494c940-4ead-447c-a1a3-e2c2193ae8df" (UID: "5494c940-4ead-447c-a1a3-e2c2193ae8df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:15.471834 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:15.471811    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5494c940-4ead-447c-a1a3-e2c2193ae8df-kube-api-access-f9qbm" (OuterVolumeSpecName: "kube-api-access-f9qbm") pod "5494c940-4ead-447c-a1a3-e2c2193ae8df" (UID: "5494c940-4ead-447c-a1a3-e2c2193ae8df"). InnerVolumeSpecName "kube-api-access-f9qbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:05:15.473927 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:15.473906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-util" (OuterVolumeSpecName: "util") pod "5494c940-4ead-447c-a1a3-e2c2193ae8df" (UID: "5494c940-4ead-447c-a1a3-e2c2193ae8df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:05:15.570329 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:15.570246    2563 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:15.570329 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:15.570278    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9qbm\" (UniqueName: \"kubernetes.io/projected/5494c940-4ead-447c-a1a3-e2c2193ae8df-kube-api-access-f9qbm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:15.570329 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:15.570290    2563 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5494c940-4ead-447c-a1a3-e2c2193ae8df-util\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:05:16.311577 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:16.311550    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8"
Sep 18 14:05:16.311577 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:16.311569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cpddt8" event={"ID":"5494c940-4ead-447c-a1a3-e2c2193ae8df","Type":"ContainerDied","Data":"7a5d8e382c254ca9895905f5ef8e934b65844c4c79d879e75f63a84fade72a9d"}
Sep 18 14:05:16.311740 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:16.311615    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a5d8e382c254ca9895905f5ef8e934b65844c4c79d879e75f63a84fade72a9d"
Sep 18 14:05:34.923913 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.923877    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"]
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924217    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1701344e-60a8-496a-be8c-bd4d9db4eee2" containerName="pull"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924232    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1701344e-60a8-496a-be8c-bd4d9db4eee2" containerName="pull"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924242    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1701344e-60a8-496a-be8c-bd4d9db4eee2" containerName="extract"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924248    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1701344e-60a8-496a-be8c-bd4d9db4eee2" containerName="extract"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924255    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" containerName="pull"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924260    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" containerName="pull"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924271    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfaf315b-d1e8-45b6-b99b-14d16e01a624" containerName="pull"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924276    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfaf315b-d1e8-45b6-b99b-14d16e01a624" containerName="pull"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924285    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" containerName="util"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924292    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" containerName="util"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924298    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" containerName="extract"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924306    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" containerName="extract"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924322    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5494c940-4ead-447c-a1a3-e2c2193ae8df" containerName="pull"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924329    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5494c940-4ead-447c-a1a3-e2c2193ae8df" containerName="pull"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924344    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5494c940-4ead-447c-a1a3-e2c2193ae8df" containerName="util"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924352    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5494c940-4ead-447c-a1a3-e2c2193ae8df" containerName="util"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924363    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5494c940-4ead-447c-a1a3-e2c2193ae8df" containerName="extract"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924371    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5494c940-4ead-447c-a1a3-e2c2193ae8df" containerName="extract"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924384    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfaf315b-d1e8-45b6-b99b-14d16e01a624" containerName="util"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924393    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfaf315b-d1e8-45b6-b99b-14d16e01a624" containerName="util"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924401    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1701344e-60a8-496a-be8c-bd4d9db4eee2" containerName="util"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924409    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1701344e-60a8-496a-be8c-bd4d9db4eee2" containerName="util"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924417    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfaf315b-d1e8-45b6-b99b-14d16e01a624" containerName="extract"
Sep 18 14:05:34.924433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924422    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfaf315b-d1e8-45b6-b99b-14d16e01a624" containerName="extract"
Sep 18 14:05:34.925265 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924502    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1701344e-60a8-496a-be8c-bd4d9db4eee2" containerName="extract"
Sep 18 14:05:34.925265 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924516    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6841c85b-6f9d-41a7-a2fe-cf9ddf0036fb" containerName="extract"
Sep 18 14:05:34.925265 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924527    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5494c940-4ead-447c-a1a3-e2c2193ae8df" containerName="extract"
Sep 18 14:05:34.925265 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.924536    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfaf315b-d1e8-45b6-b99b-14d16e01a624" containerName="extract"
Sep 18 14:05:34.927231 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.927212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:34.929802 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.929779    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 14:05:34.929959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.929858    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 14:05:34.929959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.929871    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tm7c4\""
Sep 18 14:05:34.930651 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.930633    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 14:05:34.930755 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.930654    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 14:05:34.935751 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:34.935344    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"]
Sep 18 14:05:35.037373 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:35.037331    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mmb\" (UniqueName: \"kubernetes.io/projected/14aa193a-7901-4581-9ba3-3b110c268113-kube-api-access-47mmb\") pod \"kuadrant-console-plugin-fbdd4f445-hnssn\" (UID: \"14aa193a-7901-4581-9ba3-3b110c268113\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:35.037556 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:35.037400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/14aa193a-7901-4581-9ba3-3b110c268113-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hnssn\" (UID: \"14aa193a-7901-4581-9ba3-3b110c268113\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:35.037556 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:35.037442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/14aa193a-7901-4581-9ba3-3b110c268113-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hnssn\" (UID: \"14aa193a-7901-4581-9ba3-3b110c268113\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:35.138704 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:35.138661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/14aa193a-7901-4581-9ba3-3b110c268113-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hnssn\" (UID: \"14aa193a-7901-4581-9ba3-3b110c268113\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:35.138899 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:35.138749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mmb\" (UniqueName: \"kubernetes.io/projected/14aa193a-7901-4581-9ba3-3b110c268113-kube-api-access-47mmb\") pod \"kuadrant-console-plugin-fbdd4f445-hnssn\" (UID: \"14aa193a-7901-4581-9ba3-3b110c268113\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:35.138899 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:35.138787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/14aa193a-7901-4581-9ba3-3b110c268113-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hnssn\" (UID: \"14aa193a-7901-4581-9ba3-3b110c268113\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:35.138899 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:05:35.138834    2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 18 14:05:35.139068 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:05:35.138911    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14aa193a-7901-4581-9ba3-3b110c268113-plugin-serving-cert podName:14aa193a-7901-4581-9ba3-3b110c268113 nodeName:}" failed. No retries permitted until 2026-09-18 14:05:35.63888906 +0000 UTC m=+423.795409334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/14aa193a-7901-4581-9ba3-3b110c268113-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-hnssn" (UID: "14aa193a-7901-4581-9ba3-3b110c268113") : secret "plugin-serving-cert" not found
Sep 18 14:05:35.139495 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:35.139473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/14aa193a-7901-4581-9ba3-3b110c268113-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hnssn\" (UID: \"14aa193a-7901-4581-9ba3-3b110c268113\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:35.149538 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:35.149509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mmb\" (UniqueName: \"kubernetes.io/projected/14aa193a-7901-4581-9ba3-3b110c268113-kube-api-access-47mmb\") pod \"kuadrant-console-plugin-fbdd4f445-hnssn\" (UID: \"14aa193a-7901-4581-9ba3-3b110c268113\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:35.644180 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:35.644118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/14aa193a-7901-4581-9ba3-3b110c268113-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hnssn\" (UID: \"14aa193a-7901-4581-9ba3-3b110c268113\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:35.646573 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:35.646544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/14aa193a-7901-4581-9ba3-3b110c268113-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hnssn\" (UID: \"14aa193a-7901-4581-9ba3-3b110c268113\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:35.838440 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:35.838407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"
Sep 18 14:05:36.119381 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:36.119348    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn"]
Sep 18 14:05:36.121298 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:05:36.121267    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14aa193a_7901_4581_9ba3_3b110c268113.slice/crio-347b50be9cc239f3957af23ec982cc33732ab68ebd3436a4417ef9d51c6d43d3 WatchSource:0}: Error finding container 347b50be9cc239f3957af23ec982cc33732ab68ebd3436a4417ef9d51c6d43d3: Status 404 returned error can't find the container with id 347b50be9cc239f3957af23ec982cc33732ab68ebd3436a4417ef9d51c6d43d3
Sep 18 14:05:36.415798 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:36.415714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn" event={"ID":"14aa193a-7901-4581-9ba3-3b110c268113","Type":"ContainerStarted","Data":"347b50be9cc239f3957af23ec982cc33732ab68ebd3436a4417ef9d51c6d43d3"}
Sep 18 14:05:42.449140 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:42.449090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn" event={"ID":"14aa193a-7901-4581-9ba3-3b110c268113","Type":"ContainerStarted","Data":"000eac818539f6861dab139ae81bfc704f8d2c4d4f20b8c67cf2b63112aa14c0"}
Sep 18 14:05:42.464590 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:05:42.464539    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hnssn" podStartSLOduration=3.02991563 podStartE2EDuration="8.464521397s" totalImagesPullingTime="5.434605767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:05:42.464252652 +0000 UTC m=+430.620772934" watchObservedRunningTime="2026-09-18 14:05:42.464521397 +0000 UTC m=+430.621041692"
Sep 18 14:06:20.427912 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.427871    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:06:20.449104 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.449065    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:06:20.449268 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.449177    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"
Sep 18 14:06:20.452282 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.452251    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 14:06:20.457469 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.457442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:06:20.488947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.488901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/853bb56f-f351-4065-a812-bac5160b5e34-config-file\") pod \"limitador-limitador-84994f7ffc-ptvsj\" (UID: \"853bb56f-f351-4065-a812-bac5160b5e34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"
Sep 18 14:06:20.489469 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.489344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrwpp\" (UniqueName: \"kubernetes.io/projected/853bb56f-f351-4065-a812-bac5160b5e34-kube-api-access-qrwpp\") pod \"limitador-limitador-84994f7ffc-ptvsj\" (UID: \"853bb56f-f351-4065-a812-bac5160b5e34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"
Sep 18 14:06:20.589805 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.589762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrwpp\" (UniqueName: \"kubernetes.io/projected/853bb56f-f351-4065-a812-bac5160b5e34-kube-api-access-qrwpp\") pod \"limitador-limitador-84994f7ffc-ptvsj\" (UID: \"853bb56f-f351-4065-a812-bac5160b5e34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"
Sep 18 14:06:20.590012 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.589853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/853bb56f-f351-4065-a812-bac5160b5e34-config-file\") pod \"limitador-limitador-84994f7ffc-ptvsj\" (UID: \"853bb56f-f351-4065-a812-bac5160b5e34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"
Sep 18 14:06:20.590591 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.590566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/853bb56f-f351-4065-a812-bac5160b5e34-config-file\") pod \"limitador-limitador-84994f7ffc-ptvsj\" (UID: \"853bb56f-f351-4065-a812-bac5160b5e34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"
Sep 18 14:06:20.597851 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.597822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrwpp\" (UniqueName: \"kubernetes.io/projected/853bb56f-f351-4065-a812-bac5160b5e34-kube-api-access-qrwpp\") pod \"limitador-limitador-84994f7ffc-ptvsj\" (UID: \"853bb56f-f351-4065-a812-bac5160b5e34\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"
Sep 18 14:06:20.762559 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.762463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"
Sep 18 14:06:20.849628 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.849544    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p6pld"]
Sep 18 14:06:20.853474 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.853454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld"
Sep 18 14:06:20.855677 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.855656    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cbm25\""
Sep 18 14:06:20.860793 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.860626    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p6pld"]
Sep 18 14:06:20.891806 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.891768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kndhq\" (UniqueName: \"kubernetes.io/projected/61c7c118-7e6e-45ad-8adc-f30090687f09-kube-api-access-kndhq\") pod \"authorino-7bcfd8f9cd-p6pld\" (UID: \"61c7c118-7e6e-45ad-8adc-f30090687f09\") " pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld"
Sep 18 14:06:20.992570 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:20.992529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kndhq\" (UniqueName: \"kubernetes.io/projected/61c7c118-7e6e-45ad-8adc-f30090687f09-kube-api-access-kndhq\") pod \"authorino-7bcfd8f9cd-p6pld\" (UID: \"61c7c118-7e6e-45ad-8adc-f30090687f09\") " pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld"
Sep 18 14:06:21.004997 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:21.004964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kndhq\" (UniqueName: \"kubernetes.io/projected/61c7c118-7e6e-45ad-8adc-f30090687f09-kube-api-access-kndhq\") pod \"authorino-7bcfd8f9cd-p6pld\" (UID: \"61c7c118-7e6e-45ad-8adc-f30090687f09\") " pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld"
Sep 18 14:06:21.036947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:21.036912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:06:21.038232 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:06:21.038196    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853bb56f_f351_4065_a812_bac5160b5e34.slice/crio-550ab19a19759ac73fb999092ced078929ea4910d13ba164f5f199d0a1d5f488 WatchSource:0}: Error finding container 550ab19a19759ac73fb999092ced078929ea4910d13ba164f5f199d0a1d5f488: Status 404 returned error can't find the container with id 550ab19a19759ac73fb999092ced078929ea4910d13ba164f5f199d0a1d5f488
Sep 18 14:06:21.163770 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:21.163735    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld"
Sep 18 14:06:21.434956 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:21.434931    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p6pld"]
Sep 18 14:06:21.436485 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:06:21.436456    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61c7c118_7e6e_45ad_8adc_f30090687f09.slice/crio-a23a5be59314968d0154e42ab02c67152c66987c90d2091195837e3779f5c730 WatchSource:0}: Error finding container a23a5be59314968d0154e42ab02c67152c66987c90d2091195837e3779f5c730: Status 404 returned error can't find the container with id a23a5be59314968d0154e42ab02c67152c66987c90d2091195837e3779f5c730
Sep 18 14:06:21.647998 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:21.647960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj" event={"ID":"853bb56f-f351-4065-a812-bac5160b5e34","Type":"ContainerStarted","Data":"550ab19a19759ac73fb999092ced078929ea4910d13ba164f5f199d0a1d5f488"}
Sep 18 14:06:21.649351 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:21.649324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld" event={"ID":"61c7c118-7e6e-45ad-8adc-f30090687f09","Type":"ContainerStarted","Data":"a23a5be59314968d0154e42ab02c67152c66987c90d2091195837e3779f5c730"}
Sep 18 14:06:23.845200 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:23.845168    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p6pld"]
Sep 18 14:06:24.670513 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:24.670476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj" event={"ID":"853bb56f-f351-4065-a812-bac5160b5e34","Type":"ContainerStarted","Data":"2ab5a3520b2b4432aa14346b522b3a241e91f5be5939b1927dbdd9d010b86ab3"}
Sep 18 14:06:24.670711 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:24.670598    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"
Sep 18 14:06:24.673171 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:24.673142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld" event={"ID":"61c7c118-7e6e-45ad-8adc-f30090687f09","Type":"ContainerStarted","Data":"5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085"}
Sep 18 14:06:24.673297 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:24.673235    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld" podUID="61c7c118-7e6e-45ad-8adc-f30090687f09" containerName="authorino" containerID="cri-o://5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085" gracePeriod=30
Sep 18 14:06:24.689995 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:24.689928    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj" podStartSLOduration=1.833360607 podStartE2EDuration="4.689909453s" totalImagesPullingTime="2.856548846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:06:24.685770162 +0000 UTC m=+472.842290442" watchObservedRunningTime="2026-09-18 14:06:24.689909453 +0000 UTC m=+472.846429749"
Sep 18 14:06:24.700609 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:24.700566    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld" podStartSLOduration=2.235890397 podStartE2EDuration="4.700552836s" totalImagesPullingTime="2.464662439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:06:24.699415327 +0000 UTC m=+472.855935609" watchObservedRunningTime="2026-09-18 14:06:24.700552836 +0000 UTC m=+472.857073116"
Sep 18 14:06:24.923456 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:24.923433    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld"
Sep 18 14:06:25.030240 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.030211    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kndhq\" (UniqueName: \"kubernetes.io/projected/61c7c118-7e6e-45ad-8adc-f30090687f09-kube-api-access-kndhq\") pod \"61c7c118-7e6e-45ad-8adc-f30090687f09\" (UID: \"61c7c118-7e6e-45ad-8adc-f30090687f09\") "
Sep 18 14:06:25.032388 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.032364    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c7c118-7e6e-45ad-8adc-f30090687f09-kube-api-access-kndhq" (OuterVolumeSpecName: "kube-api-access-kndhq") pod "61c7c118-7e6e-45ad-8adc-f30090687f09" (UID: "61c7c118-7e6e-45ad-8adc-f30090687f09"). InnerVolumeSpecName "kube-api-access-kndhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:06:25.131349 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.131319    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kndhq\" (UniqueName: \"kubernetes.io/projected/61c7c118-7e6e-45ad-8adc-f30090687f09-kube-api-access-kndhq\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:06:25.680701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.680664    2563 generic.go:356] "Generic (PLEG): container finished" podID="61c7c118-7e6e-45ad-8adc-f30090687f09" containerID="5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085" exitCode=0
Sep 18 14:06:25.680877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.680708    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld"
Sep 18 14:06:25.680877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.680743    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld" event={"ID":"61c7c118-7e6e-45ad-8adc-f30090687f09","Type":"ContainerDied","Data":"5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085"}
Sep 18 14:06:25.680877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.680786    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-p6pld" event={"ID":"61c7c118-7e6e-45ad-8adc-f30090687f09","Type":"ContainerDied","Data":"a23a5be59314968d0154e42ab02c67152c66987c90d2091195837e3779f5c730"}
Sep 18 14:06:25.680877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.680805    2563 scope.go:117] "RemoveContainer" containerID="5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085"
Sep 18 14:06:25.690225 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.690203    2563 scope.go:117] "RemoveContainer" containerID="5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085"
Sep 18 14:06:25.690569 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:06:25.690550    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085\": container with ID starting with 5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085 not found: ID does not exist" containerID="5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085"
Sep 18 14:06:25.690615 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.690579    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085"} err="failed to get container status \"5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085\": rpc error: code = NotFound desc = could not find container \"5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085\": container with ID starting with 5bebe1c7f700279c70639834b130f43cbbab73b12c79ae18219c9b41295f6085 not found: ID does not exist"
Sep 18 14:06:25.703744 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.703718    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p6pld"]
Sep 18 14:06:25.707956 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:25.707930    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-p6pld"]
Sep 18 14:06:26.287740 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:26.287706    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61c7c118-7e6e-45ad-8adc-f30090687f09" path="/var/lib/kubelet/pods/61c7c118-7e6e-45ad-8adc-f30090687f09/volumes"
Sep 18 14:06:35.682465 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:35.682432    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"
Sep 18 14:06:54.793285 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:54.793244    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-lpc75"]
Sep 18 14:06:54.793862 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:54.793638    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61c7c118-7e6e-45ad-8adc-f30090687f09" containerName="authorino"
Sep 18 14:06:54.793862 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:54.793653    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="61c7c118-7e6e-45ad-8adc-f30090687f09" containerName="authorino"
Sep 18 14:06:54.793862 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:54.793724    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="61c7c118-7e6e-45ad-8adc-f30090687f09" containerName="authorino"
Sep 18 14:06:54.799371 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:54.799345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lpc75"
Sep 18 14:06:54.801955 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:54.801928    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cbm25\""
Sep 18 14:06:54.804557 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:54.804531    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-lpc75"]
Sep 18 14:06:54.886931 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:54.886899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq7rf\" (UniqueName: \"kubernetes.io/projected/a3e8c0be-f6a7-44c5-9100-0823598aa87c-kube-api-access-zq7rf\") pod \"authorino-557f876f69-lpc75\" (UID: \"a3e8c0be-f6a7-44c5-9100-0823598aa87c\") " pod="kuadrant-system/authorino-557f876f69-lpc75"
Sep 18 14:06:54.988322 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:54.988282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq7rf\" (UniqueName: \"kubernetes.io/projected/a3e8c0be-f6a7-44c5-9100-0823598aa87c-kube-api-access-zq7rf\") pod \"authorino-557f876f69-lpc75\" (UID: \"a3e8c0be-f6a7-44c5-9100-0823598aa87c\") " pod="kuadrant-system/authorino-557f876f69-lpc75"
Sep 18 14:06:54.999652 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:54.999622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq7rf\" (UniqueName: \"kubernetes.io/projected/a3e8c0be-f6a7-44c5-9100-0823598aa87c-kube-api-access-zq7rf\") pod \"authorino-557f876f69-lpc75\" (UID: \"a3e8c0be-f6a7-44c5-9100-0823598aa87c\") " pod="kuadrant-system/authorino-557f876f69-lpc75"
Sep 18 14:06:55.013438 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.013404    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-lpc75"]
Sep 18 14:06:55.013676 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.013663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lpc75"
Sep 18 14:06:55.045884 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.045807    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cd994f57d-rtsrz"]
Sep 18 14:06:55.049177 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.049153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd994f57d-rtsrz"
Sep 18 14:06:55.058386 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.058357    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cd994f57d-rtsrz"]
Sep 18 14:06:55.190744 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.190708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlp9m\" (UniqueName: \"kubernetes.io/projected/7c365e1c-6550-42b5-bcf9-df3110d52341-kube-api-access-tlp9m\") pod \"authorino-cd994f57d-rtsrz\" (UID: \"7c365e1c-6550-42b5-bcf9-df3110d52341\") " pod="kuadrant-system/authorino-cd994f57d-rtsrz"
Sep 18 14:06:55.291197 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.291160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlp9m\" (UniqueName: \"kubernetes.io/projected/7c365e1c-6550-42b5-bcf9-df3110d52341-kube-api-access-tlp9m\") pod \"authorino-cd994f57d-rtsrz\" (UID: \"7c365e1c-6550-42b5-bcf9-df3110d52341\") " pod="kuadrant-system/authorino-cd994f57d-rtsrz"
Sep 18 14:06:55.292009 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.291989    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-lpc75"]
Sep 18 14:06:55.293974 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:06:55.293955    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3e8c0be_f6a7_44c5_9100_0823598aa87c.slice/crio-c371093556a11ff84c8f446931f9ff697e0df78d889e09add41128526dd10296 WatchSource:0}: Error finding container c371093556a11ff84c8f446931f9ff697e0df78d889e09add41128526dd10296: Status 404 returned error can't find the container with id c371093556a11ff84c8f446931f9ff697e0df78d889e09add41128526dd10296
Sep 18 14:06:55.299553 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.299534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlp9m\" (UniqueName: \"kubernetes.io/projected/7c365e1c-6550-42b5-bcf9-df3110d52341-kube-api-access-tlp9m\") pod \"authorino-cd994f57d-rtsrz\" (UID: \"7c365e1c-6550-42b5-bcf9-df3110d52341\") " pod="kuadrant-system/authorino-cd994f57d-rtsrz"
Sep 18 14:06:55.362579 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.362551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd994f57d-rtsrz"
Sep 18 14:06:55.636336 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.636256    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cd994f57d-rtsrz"]
Sep 18 14:06:55.640370 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:06:55.640337    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c365e1c_6550_42b5_bcf9_df3110d52341.slice/crio-e007ff8a703f8ae20a0e655880e28083aa0a04f9694a1563b6647c91271b73c6 WatchSource:0}: Error finding container e007ff8a703f8ae20a0e655880e28083aa0a04f9694a1563b6647c91271b73c6: Status 404 returned error can't find the container with id e007ff8a703f8ae20a0e655880e28083aa0a04f9694a1563b6647c91271b73c6
Sep 18 14:06:55.838896 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.838853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd994f57d-rtsrz" event={"ID":"7c365e1c-6550-42b5-bcf9-df3110d52341","Type":"ContainerStarted","Data":"e007ff8a703f8ae20a0e655880e28083aa0a04f9694a1563b6647c91271b73c6"}
Sep 18 14:06:55.840373 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.840346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-lpc75" event={"ID":"a3e8c0be-f6a7-44c5-9100-0823598aa87c","Type":"ContainerStarted","Data":"53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14"}
Sep 18 14:06:55.840506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.840375    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-lpc75" podUID="a3e8c0be-f6a7-44c5-9100-0823598aa87c" containerName="authorino" containerID="cri-o://53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14" gracePeriod=30
Sep 18 14:06:55.840506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.840385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-lpc75" event={"ID":"a3e8c0be-f6a7-44c5-9100-0823598aa87c","Type":"ContainerStarted","Data":"c371093556a11ff84c8f446931f9ff697e0df78d889e09add41128526dd10296"}
Sep 18 14:06:55.857642 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:55.857588    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-lpc75" podStartSLOduration=1.498480582 podStartE2EDuration="1.857569444s" totalImagesPullingTime="359.088862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:06:55.855456835 +0000 UTC m=+504.011977127" watchObservedRunningTime="2026-09-18 14:06:55.857569444 +0000 UTC m=+504.014089725"
Sep 18 14:06:56.088734 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.088706    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lpc75"
Sep 18 14:06:56.198679 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.198646    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq7rf\" (UniqueName: \"kubernetes.io/projected/a3e8c0be-f6a7-44c5-9100-0823598aa87c-kube-api-access-zq7rf\") pod \"a3e8c0be-f6a7-44c5-9100-0823598aa87c\" (UID: \"a3e8c0be-f6a7-44c5-9100-0823598aa87c\") "
Sep 18 14:06:56.200661 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.200628    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e8c0be-f6a7-44c5-9100-0823598aa87c-kube-api-access-zq7rf" (OuterVolumeSpecName: "kube-api-access-zq7rf") pod "a3e8c0be-f6a7-44c5-9100-0823598aa87c" (UID: "a3e8c0be-f6a7-44c5-9100-0823598aa87c"). InnerVolumeSpecName "kube-api-access-zq7rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:06:56.299398 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.299357    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq7rf\" (UniqueName: \"kubernetes.io/projected/a3e8c0be-f6a7-44c5-9100-0823598aa87c-kube-api-access-zq7rf\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:06:56.847773 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.847735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd994f57d-rtsrz" event={"ID":"7c365e1c-6550-42b5-bcf9-df3110d52341","Type":"ContainerStarted","Data":"e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf"}
Sep 18 14:06:56.849062 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.849036    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3e8c0be-f6a7-44c5-9100-0823598aa87c" containerID="53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14" exitCode=0
Sep 18 14:06:56.849173 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.849086    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-lpc75"
Sep 18 14:06:56.849173 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.849086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-lpc75" event={"ID":"a3e8c0be-f6a7-44c5-9100-0823598aa87c","Type":"ContainerDied","Data":"53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14"}
Sep 18 14:06:56.849258 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.849180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-lpc75" event={"ID":"a3e8c0be-f6a7-44c5-9100-0823598aa87c","Type":"ContainerDied","Data":"c371093556a11ff84c8f446931f9ff697e0df78d889e09add41128526dd10296"}
Sep 18 14:06:56.849258 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.849203    2563 scope.go:117] "RemoveContainer" containerID="53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14"
Sep 18 14:06:56.857576 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.857560    2563 scope.go:117] "RemoveContainer" containerID="53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14"
Sep 18 14:06:56.857830 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:06:56.857810    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14\": container with ID starting with 53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14 not found: ID does not exist" containerID="53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14"
Sep 18 14:06:56.857893 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.857838    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14"} err="failed to get container status \"53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14\": rpc error: code = NotFound desc = could not find container \"53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14\": container with ID starting with 53f06cf7529559a6d26cb80ca5c802a57b8277f06f5a13dfe6ced497b52c1f14 not found: ID does not exist"
Sep 18 14:06:56.866463 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.866415    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-cd994f57d-rtsrz" podStartSLOduration=1.38251456 podStartE2EDuration="1.866400514s" totalImagesPullingTime="483.885954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:06:56.863014034 +0000 UTC m=+505.019534316" watchObservedRunningTime="2026-09-18 14:06:56.866400514 +0000 UTC m=+505.022920806"
Sep 18 14:06:56.878690 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.878664    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-lpc75"]
Sep 18 14:06:56.886968 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.886942    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-lpc75"]
Sep 18 14:06:56.974622 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.974591    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c85b4f584-776jm"]
Sep 18 14:06:56.974959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.974947    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e8c0be-f6a7-44c5-9100-0823598aa87c" containerName="authorino"
Sep 18 14:06:56.975003 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.974961    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e8c0be-f6a7-44c5-9100-0823598aa87c" containerName="authorino"
Sep 18 14:06:56.975044 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.975036    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e8c0be-f6a7-44c5-9100-0823598aa87c" containerName="authorino"
Sep 18 14:06:56.979244 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.979223    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c85b4f584-776jm"
Sep 18 14:06:56.981601 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.981584    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 14:06:56.991520 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:56.991498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c85b4f584-776jm"]
Sep 18 14:06:57.104908 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:57.104834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnq7r\" (UniqueName: \"kubernetes.io/projected/02b1cbb0-4415-4787-99de-75db1e93738a-kube-api-access-nnq7r\") pod \"authorino-6c85b4f584-776jm\" (UID: \"02b1cbb0-4415-4787-99de-75db1e93738a\") " pod="kuadrant-system/authorino-6c85b4f584-776jm"
Sep 18 14:06:57.104908 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:57.104904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02b1cbb0-4415-4787-99de-75db1e93738a-tls-cert\") pod \"authorino-6c85b4f584-776jm\" (UID: \"02b1cbb0-4415-4787-99de-75db1e93738a\") " pod="kuadrant-system/authorino-6c85b4f584-776jm"
Sep 18 14:06:57.206193 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:57.206124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02b1cbb0-4415-4787-99de-75db1e93738a-tls-cert\") pod \"authorino-6c85b4f584-776jm\" (UID: \"02b1cbb0-4415-4787-99de-75db1e93738a\") " pod="kuadrant-system/authorino-6c85b4f584-776jm"
Sep 18 14:06:57.206371 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:57.206221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnq7r\" (UniqueName: \"kubernetes.io/projected/02b1cbb0-4415-4787-99de-75db1e93738a-kube-api-access-nnq7r\") pod \"authorino-6c85b4f584-776jm\" (UID: \"02b1cbb0-4415-4787-99de-75db1e93738a\") " pod="kuadrant-system/authorino-6c85b4f584-776jm"
Sep 18 14:06:57.208787 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:57.208756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02b1cbb0-4415-4787-99de-75db1e93738a-tls-cert\") pod \"authorino-6c85b4f584-776jm\" (UID: \"02b1cbb0-4415-4787-99de-75db1e93738a\") " pod="kuadrant-system/authorino-6c85b4f584-776jm"
Sep 18 14:06:57.216861 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:57.216837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnq7r\" (UniqueName: \"kubernetes.io/projected/02b1cbb0-4415-4787-99de-75db1e93738a-kube-api-access-nnq7r\") pod \"authorino-6c85b4f584-776jm\" (UID: \"02b1cbb0-4415-4787-99de-75db1e93738a\") " pod="kuadrant-system/authorino-6c85b4f584-776jm"
Sep 18 14:06:57.287940 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:57.287909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c85b4f584-776jm"
Sep 18 14:06:57.562043 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:57.562016    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c85b4f584-776jm"]
Sep 18 14:06:57.563926 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:06:57.563894    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02b1cbb0_4415_4787_99de_75db1e93738a.slice/crio-47ed1ea18223bc8300788a8b374f942f6139846cf6f8f1880e7e8945f6096d60 WatchSource:0}: Error finding container 47ed1ea18223bc8300788a8b374f942f6139846cf6f8f1880e7e8945f6096d60: Status 404 returned error can't find the container with id 47ed1ea18223bc8300788a8b374f942f6139846cf6f8f1880e7e8945f6096d60
Sep 18 14:06:57.855659 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:57.855565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c85b4f584-776jm" event={"ID":"02b1cbb0-4415-4787-99de-75db1e93738a","Type":"ContainerStarted","Data":"47ed1ea18223bc8300788a8b374f942f6139846cf6f8f1880e7e8945f6096d60"}
Sep 18 14:06:58.287711 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:58.287678    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e8c0be-f6a7-44c5-9100-0823598aa87c" path="/var/lib/kubelet/pods/a3e8c0be-f6a7-44c5-9100-0823598aa87c/volumes"
Sep 18 14:06:58.863429 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:58.863389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c85b4f584-776jm" event={"ID":"02b1cbb0-4415-4787-99de-75db1e93738a","Type":"ContainerStarted","Data":"6188714d10b76f0752972ba7088ab7262796afb99ec6ac22e15f1e2d894a1ecb"}
Sep 18 14:06:58.903408 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:58.903353    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c85b4f584-776jm" podStartSLOduration=2.411246534 podStartE2EDuration="2.903336785s" totalImagesPullingTime="492.090251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:06:58.897478848 +0000 UTC m=+507.053999130" watchObservedRunningTime="2026-09-18 14:06:58.903336785 +0000 UTC m=+507.059857065"
Sep 18 14:06:58.944171 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:58.944123    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cd994f57d-rtsrz"]
Sep 18 14:06:58.944326 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:58.944305    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-cd994f57d-rtsrz" podUID="7c365e1c-6550-42b5-bcf9-df3110d52341" containerName="authorino" containerID="cri-o://e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf" gracePeriod=30
Sep 18 14:06:59.196610 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.196585    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd994f57d-rtsrz"
Sep 18 14:06:59.322684 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.322644    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlp9m\" (UniqueName: \"kubernetes.io/projected/7c365e1c-6550-42b5-bcf9-df3110d52341-kube-api-access-tlp9m\") pod \"7c365e1c-6550-42b5-bcf9-df3110d52341\" (UID: \"7c365e1c-6550-42b5-bcf9-df3110d52341\") "
Sep 18 14:06:59.324811 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.324778    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c365e1c-6550-42b5-bcf9-df3110d52341-kube-api-access-tlp9m" (OuterVolumeSpecName: "kube-api-access-tlp9m") pod "7c365e1c-6550-42b5-bcf9-df3110d52341" (UID: "7c365e1c-6550-42b5-bcf9-df3110d52341"). InnerVolumeSpecName "kube-api-access-tlp9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:06:59.423506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.423459    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlp9m\" (UniqueName: \"kubernetes.io/projected/7c365e1c-6550-42b5-bcf9-df3110d52341-kube-api-access-tlp9m\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:06:59.870444 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.870409    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c365e1c-6550-42b5-bcf9-df3110d52341" containerID="e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf" exitCode=0
Sep 18 14:06:59.870893 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.870451    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd994f57d-rtsrz"
Sep 18 14:06:59.870893 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.870492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd994f57d-rtsrz" event={"ID":"7c365e1c-6550-42b5-bcf9-df3110d52341","Type":"ContainerDied","Data":"e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf"}
Sep 18 14:06:59.870893 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.870535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd994f57d-rtsrz" event={"ID":"7c365e1c-6550-42b5-bcf9-df3110d52341","Type":"ContainerDied","Data":"e007ff8a703f8ae20a0e655880e28083aa0a04f9694a1563b6647c91271b73c6"}
Sep 18 14:06:59.870893 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.870552    2563 scope.go:117] "RemoveContainer" containerID="e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf"
Sep 18 14:06:59.883306 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.883289    2563 scope.go:117] "RemoveContainer" containerID="e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf"
Sep 18 14:06:59.883570 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:06:59.883553    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf\": container with ID starting with e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf not found: ID does not exist" containerID="e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf"
Sep 18 14:06:59.883617 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.883577    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf"} err="failed to get container status \"e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf\": rpc error: code = NotFound desc = could not find container \"e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf\": container with ID starting with e0f0360ee0d9af78a6ed62c23eaf9efc221e1a5aaa75c7662eeceea5ac6274bf not found: ID does not exist"
Sep 18 14:06:59.895253 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.895228    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cd994f57d-rtsrz"]
Sep 18 14:06:59.901411 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:06:59.901391    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-cd994f57d-rtsrz"]
Sep 18 14:07:00.287976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:07:00.287937    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c365e1c-6550-42b5-bcf9-df3110d52341" path="/var/lib/kubelet/pods/7c365e1c-6550-42b5-bcf9-df3110d52341/volumes"
Sep 18 14:07:39.716517 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:07:39.716476    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:08:18.553977 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:08:18.553938    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:08:32.613335 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:08:32.613235    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:08:32.622640 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:08:32.615163    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:08:40.429613 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:08:40.429578    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:09:17.019079 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:17.019039    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:09:26.726022 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:26.725981    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:09:34.934694 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:34.934656    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"]
Sep 18 14:09:34.935501 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:34.935161    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c365e1c-6550-42b5-bcf9-df3110d52341" containerName="authorino"
Sep 18 14:09:34.935501 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:34.935182    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c365e1c-6550-42b5-bcf9-df3110d52341" containerName="authorino"
Sep 18 14:09:34.935501 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:34.935314    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c365e1c-6550-42b5-bcf9-df3110d52341" containerName="authorino"
Sep 18 14:09:34.938342 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:34.938319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:34.941174 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:34.941151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 14:09:34.942283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:34.942265    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 14:09:34.942361 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:34.942276    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5zz6x\""
Sep 18 14:09:34.942361 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:34.942268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 18 14:09:34.952277 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:34.952255    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"]
Sep 18 14:09:35.025739 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.025716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.025834 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.025764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88eea5fd-f516-4385-aba3-e7d934283539-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.025834 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.025784    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.025834 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.025821    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58x5w\" (UniqueName: \"kubernetes.io/projected/88eea5fd-f516-4385-aba3-e7d934283539-kube-api-access-58x5w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.025959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.025877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.025959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.025904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.127162 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.127123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.127263 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.127170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.127313 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.127289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.127364 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.127347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88eea5fd-f516-4385-aba3-e7d934283539-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.127418 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.127368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.127418 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.127392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58x5w\" (UniqueName: \"kubernetes.io/projected/88eea5fd-f516-4385-aba3-e7d934283539-kube-api-access-58x5w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.127536 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.127519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.127614 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.127591    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.127759 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.127726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.129597 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.129579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88eea5fd-f516-4385-aba3-e7d934283539-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.129978 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.129956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88eea5fd-f516-4385-aba3-e7d934283539-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.136612 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.136587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58x5w\" (UniqueName: \"kubernetes.io/projected/88eea5fd-f516-4385-aba3-e7d934283539-kube-api-access-58x5w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf\" (UID: \"88eea5fd-f516-4385-aba3-e7d934283539\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.248969 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.248888    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:35.549120 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.549093    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"]
Sep 18 14:09:35.551389 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:09:35.551357    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88eea5fd_f516_4385_aba3_e7d934283539.slice/crio-0b0f8ab7af91962222756f40ea32856b50ee62d19decd123ff263dabb9653a99 WatchSource:0}: Error finding container 0b0f8ab7af91962222756f40ea32856b50ee62d19decd123ff263dabb9653a99: Status 404 returned error can't find the container with id 0b0f8ab7af91962222756f40ea32856b50ee62d19decd123ff263dabb9653a99
Sep 18 14:09:35.552740 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.552720    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:09:35.728664 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:35.728627    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf" event={"ID":"88eea5fd-f516-4385-aba3-e7d934283539","Type":"ContainerStarted","Data":"0b0f8ab7af91962222756f40ea32856b50ee62d19decd123ff263dabb9653a99"}
Sep 18 14:09:36.624728 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:36.624697    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:09:36.738714 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:36.738673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf" event={"ID":"88eea5fd-f516-4385-aba3-e7d934283539","Type":"ContainerStarted","Data":"098ae6005bc302051d43a7d50ff3d21fb30d00995906ccfc1daf168f8391775f"}
Sep 18 14:09:36.762762 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:36.762698    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf" podStartSLOduration=1.720828446 podStartE2EDuration="2.762680288s" totalImagesPullingTime="1.041851842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:09:36.761015103 +0000 UTC m=+664.917535385" watchObservedRunningTime="2026-09-18 14:09:36.762680288 +0000 UTC m=+664.919200569"
Sep 18 14:09:43.418534 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:43.418492    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:09:45.249355 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:45.249313    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:45.249790 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:45.249399    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:45.262179 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:45.262144    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:45.803488 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:45.803450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf"
Sep 18 14:09:51.554652 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.554619    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"]
Sep 18 14:09:51.558170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.558150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.568406 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.568382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.568499 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.568420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.568499 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.568445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab3de34a-f278-452a-82ba-6d9c4224670d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.568576 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.568509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbrv\" (UniqueName: \"kubernetes.io/projected/ab3de34a-f278-452a-82ba-6d9c4224670d-kube-api-access-8zbrv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.568576 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.568558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.568640 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.568588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.573651 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.573634    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 14:09:51.596735 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.596710    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"]
Sep 18 14:09:51.668907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.668886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.668998 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.668920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.668998 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.668952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.668998 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.668990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.669208 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.669027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab3de34a-f278-452a-82ba-6d9c4224670d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.669208 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.669054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbrv\" (UniqueName: \"kubernetes.io/projected/ab3de34a-f278-452a-82ba-6d9c4224670d-kube-api-access-8zbrv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.669344 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.669323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.669404 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.669384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.669478 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.669456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.671085 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.671063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab3de34a-f278-452a-82ba-6d9c4224670d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.671424 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.671406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab3de34a-f278-452a-82ba-6d9c4224670d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.684845 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.684820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbrv\" (UniqueName: \"kubernetes.io/projected/ab3de34a-f278-452a-82ba-6d9c4224670d-kube-api-access-8zbrv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r\" (UID: \"ab3de34a-f278-452a-82ba-6d9c4224670d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:51.867389 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:51.867332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:09:52.141817 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:52.141791    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"]
Sep 18 14:09:52.142986 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:09:52.142962    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab3de34a_f278_452a_82ba_6d9c4224670d.slice/crio-fbd449becc9b826df30b7132259f2a7b6159d3f7f0d5c29e682275c0e260d521 WatchSource:0}: Error finding container fbd449becc9b826df30b7132259f2a7b6159d3f7f0d5c29e682275c0e260d521: Status 404 returned error can't find the container with id fbd449becc9b826df30b7132259f2a7b6159d3f7f0d5c29e682275c0e260d521
Sep 18 14:09:52.829621 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:52.829583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r" event={"ID":"ab3de34a-f278-452a-82ba-6d9c4224670d","Type":"ContainerStarted","Data":"7681157d2ea9137a6a83ee4607273a0a85e9d69356005efa3aaf2c0cf7c09866"}
Sep 18 14:09:52.829995 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:52.829633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r" event={"ID":"ab3de34a-f278-452a-82ba-6d9c4224670d","Type":"ContainerStarted","Data":"fbd449becc9b826df30b7132259f2a7b6159d3f7f0d5c29e682275c0e260d521"}
Sep 18 14:09:52.862790 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:52.862731    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r" podStartSLOduration=1.6697624960000002 podStartE2EDuration="1.862715804s" totalImagesPullingTime="192.953308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:09:52.862222233 +0000 UTC m=+681.018742514" watchObservedRunningTime="2026-09-18 14:09:52.862715804 +0000 UTC m=+681.019236075"
Sep 18 14:09:54.049703 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:09:54.049663    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:10:01.867930 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:01.867889    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:10:01.868456 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:01.867942    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:10:01.880865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:01.880836    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:10:01.892121 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:01.892096    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r"
Sep 18 14:10:56.990922 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:56.990887    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c85b4f584-776jm"]
Sep 18 14:10:56.991575 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:56.991071    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6c85b4f584-776jm" podUID="02b1cbb0-4415-4787-99de-75db1e93738a" containerName="authorino" containerID="cri-o://6188714d10b76f0752972ba7088ab7262796afb99ec6ac22e15f1e2d894a1ecb" gracePeriod=30
Sep 18 14:10:57.193687 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:57.193655    2563 generic.go:356] "Generic (PLEG): container finished" podID="02b1cbb0-4415-4787-99de-75db1e93738a" containerID="6188714d10b76f0752972ba7088ab7262796afb99ec6ac22e15f1e2d894a1ecb" exitCode=0
Sep 18 14:10:57.193810 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:57.193727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c85b4f584-776jm" event={"ID":"02b1cbb0-4415-4787-99de-75db1e93738a","Type":"ContainerDied","Data":"6188714d10b76f0752972ba7088ab7262796afb99ec6ac22e15f1e2d894a1ecb"}
Sep 18 14:10:57.234522 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:57.234501    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c85b4f584-776jm"
Sep 18 14:10:57.299458 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:57.299398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnq7r\" (UniqueName: \"kubernetes.io/projected/02b1cbb0-4415-4787-99de-75db1e93738a-kube-api-access-nnq7r\") pod \"02b1cbb0-4415-4787-99de-75db1e93738a\" (UID: \"02b1cbb0-4415-4787-99de-75db1e93738a\") "
Sep 18 14:10:57.299578 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:57.299493    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02b1cbb0-4415-4787-99de-75db1e93738a-tls-cert\") pod \"02b1cbb0-4415-4787-99de-75db1e93738a\" (UID: \"02b1cbb0-4415-4787-99de-75db1e93738a\") "
Sep 18 14:10:57.301402 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:57.301376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b1cbb0-4415-4787-99de-75db1e93738a-kube-api-access-nnq7r" (OuterVolumeSpecName: "kube-api-access-nnq7r") pod "02b1cbb0-4415-4787-99de-75db1e93738a" (UID: "02b1cbb0-4415-4787-99de-75db1e93738a"). InnerVolumeSpecName "kube-api-access-nnq7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:10:57.309115 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:57.309091    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b1cbb0-4415-4787-99de-75db1e93738a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "02b1cbb0-4415-4787-99de-75db1e93738a" (UID: "02b1cbb0-4415-4787-99de-75db1e93738a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:10:57.400238 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:57.400211    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02b1cbb0-4415-4787-99de-75db1e93738a-tls-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:10:57.400348 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:57.400239    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnq7r\" (UniqueName: \"kubernetes.io/projected/02b1cbb0-4415-4787-99de-75db1e93738a-kube-api-access-nnq7r\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:10:58.200543 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:58.200517    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c85b4f584-776jm"
Sep 18 14:10:58.200990 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:58.200558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c85b4f584-776jm" event={"ID":"02b1cbb0-4415-4787-99de-75db1e93738a","Type":"ContainerDied","Data":"47ed1ea18223bc8300788a8b374f942f6139846cf6f8f1880e7e8945f6096d60"}
Sep 18 14:10:58.200990 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:58.200607    2563 scope.go:117] "RemoveContainer" containerID="6188714d10b76f0752972ba7088ab7262796afb99ec6ac22e15f1e2d894a1ecb"
Sep 18 14:10:58.221094 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:58.221071    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c85b4f584-776jm"]
Sep 18 14:10:58.229478 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:58.229457    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c85b4f584-776jm"]
Sep 18 14:10:58.287982 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:10:58.287951    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02b1cbb0-4415-4787-99de-75db1e93738a" path="/var/lib/kubelet/pods/02b1cbb0-4415-4787-99de-75db1e93738a/volumes"
Sep 18 14:11:18.041323 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.041283    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7770-579034"]
Sep 18 14:11:18.041744 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.041628    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02b1cbb0-4415-4787-99de-75db1e93738a" containerName="authorino"
Sep 18 14:11:18.041744 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.041640    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02b1cbb0-4415-4787-99de-75db1e93738a" containerName="authorino"
Sep 18 14:11:18.041744 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.041711    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="02b1cbb0-4415-4787-99de-75db1e93738a" containerName="authorino"
Sep 18 14:11:18.046043 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.046026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-579034"
Sep 18 14:11:18.049070 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.049042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r4cmx\""
Sep 18 14:11:18.051214 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.051191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7770-579034"]
Sep 18 14:11:18.171864 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.171826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qr2\" (UniqueName: \"kubernetes.io/projected/ede154db-71c0-4c7b-ad9c-e722cc2284ab-kube-api-access-s5qr2\") pod \"test-curl-7770-579034\" (UID: \"ede154db-71c0-4c7b-ad9c-e722cc2284ab\") " pod="opendatahub/test-curl-7770-579034"
Sep 18 14:11:18.272997 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.272942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qr2\" (UniqueName: \"kubernetes.io/projected/ede154db-71c0-4c7b-ad9c-e722cc2284ab-kube-api-access-s5qr2\") pod \"test-curl-7770-579034\" (UID: \"ede154db-71c0-4c7b-ad9c-e722cc2284ab\") " pod="opendatahub/test-curl-7770-579034"
Sep 18 14:11:18.281363 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.281331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qr2\" (UniqueName: \"kubernetes.io/projected/ede154db-71c0-4c7b-ad9c-e722cc2284ab-kube-api-access-s5qr2\") pod \"test-curl-7770-579034\" (UID: \"ede154db-71c0-4c7b-ad9c-e722cc2284ab\") " pod="opendatahub/test-curl-7770-579034"
Sep 18 14:11:18.356755 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.356669    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-579034"
Sep 18 14:11:18.633497 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:18.633474    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7770-579034"]
Sep 18 14:11:18.635387 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:11:18.635362    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podede154db_71c0_4c7b_ad9c_e722cc2284ab.slice/crio-28c612c2713276f73141f2aaa65216c663b47b01e4d700ed2a7f367b7f38ef5b WatchSource:0}: Error finding container 28c612c2713276f73141f2aaa65216c663b47b01e4d700ed2a7f367b7f38ef5b: Status 404 returned error can't find the container with id 28c612c2713276f73141f2aaa65216c663b47b01e4d700ed2a7f367b7f38ef5b
Sep 18 14:11:19.316014 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:19.315974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7770-579034" event={"ID":"ede154db-71c0-4c7b-ad9c-e722cc2284ab","Type":"ContainerStarted","Data":"28c612c2713276f73141f2aaa65216c663b47b01e4d700ed2a7f367b7f38ef5b"}
Sep 18 14:11:20.367243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:20.367203    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:11:21.331046 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:21.331005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7770-579034" event={"ID":"ede154db-71c0-4c7b-ad9c-e722cc2284ab","Type":"ContainerStarted","Data":"daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373"}
Sep 18 14:11:21.361350 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:21.361280    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7770-579034" podStartSLOduration=1.417388303 podStartE2EDuration="3.361258365s" totalImagesPullingTime="1.943870062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:21.358314417 +0000 UTC m=+769.514834701" watchObservedRunningTime="2026-09-18 14:11:21.361258365 +0000 UTC m=+769.517778643"
Sep 18 14:11:22.114748 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.114701    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7770-579034"]
Sep 18 14:11:22.119406 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.119330    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7770-579034"]
Sep 18 14:11:22.226435 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.226401    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"]
Sep 18 14:11:22.226789 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.226777    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ede154db-71c0-4c7b-ad9c-e722cc2284ab" containerName="test-curl-7770-579034"
Sep 18 14:11:22.226833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.226791    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ede154db-71c0-4c7b-ad9c-e722cc2284ab" containerName="test-curl-7770-579034"
Sep 18 14:11:22.226868 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.226847    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ede154db-71c0-4c7b-ad9c-e722cc2284ab" containerName="test-curl-7770-579034"
Sep 18 14:11:22.230007 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.229989    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:11:22.234092 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.234074    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-4d9222-dockercfg-2jpks\""
Sep 18 14:11:22.234215 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.234122    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-4d9222\""
Sep 18 14:11:22.243768 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.243735    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7770-82482c"]
Sep 18 14:11:22.247822 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.247792    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"]
Sep 18 14:11:22.247947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.247891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-82482c"
Sep 18 14:11:22.259852 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.259819    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7770-82482c"]
Sep 18 14:11:22.311189 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.311151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbh8\" (UniqueName: \"kubernetes.io/projected/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-kube-api-access-6mbh8\") pod \"payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx\" (UID: \"dcaf20b7-f9d7-45bd-8dd1-57d79812916f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:11:22.311372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.311244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx\" (UID: \"dcaf20b7-f9d7-45bd-8dd1-57d79812916f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:11:22.337368 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:11:22.337332    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7770-579034" container="test-curl-7770-579034"
Sep 18 14:11:22.412354 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.412264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbh8\" (UniqueName: \"kubernetes.io/projected/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-kube-api-access-6mbh8\") pod \"payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx\" (UID: \"dcaf20b7-f9d7-45bd-8dd1-57d79812916f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:11:22.412354 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.412326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx\" (UID: \"dcaf20b7-f9d7-45bd-8dd1-57d79812916f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:11:22.412558 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.412398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvng\" (UniqueName: \"kubernetes.io/projected/5202ebe1-2a17-471b-8d63-deaf71e2fef0-kube-api-access-2jvng\") pod \"test-curl-7770-82482c\" (UID: \"5202ebe1-2a17-471b-8d63-deaf71e2fef0\") " pod="opendatahub/test-curl-7770-82482c"
Sep 18 14:11:22.413022 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.413001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx\" (UID: \"dcaf20b7-f9d7-45bd-8dd1-57d79812916f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:11:22.450089 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.450058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbh8\" (UniqueName: \"kubernetes.io/projected/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-kube-api-access-6mbh8\") pod \"payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx\" (UID: \"dcaf20b7-f9d7-45bd-8dd1-57d79812916f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:11:22.513505 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.513470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvng\" (UniqueName: \"kubernetes.io/projected/5202ebe1-2a17-471b-8d63-deaf71e2fef0-kube-api-access-2jvng\") pod \"test-curl-7770-82482c\" (UID: \"5202ebe1-2a17-471b-8d63-deaf71e2fef0\") " pod="opendatahub/test-curl-7770-82482c"
Sep 18 14:11:22.540022 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.539990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:11:22.547560 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.547532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvng\" (UniqueName: \"kubernetes.io/projected/5202ebe1-2a17-471b-8d63-deaf71e2fef0-kube-api-access-2jvng\") pod \"test-curl-7770-82482c\" (UID: \"5202ebe1-2a17-471b-8d63-deaf71e2fef0\") " pod="opendatahub/test-curl-7770-82482c"
Sep 18 14:11:22.557341 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.557314    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-82482c"
Sep 18 14:11:22.827760 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.827727    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"]
Sep 18 14:11:22.829140 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:11:22.829103    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcaf20b7_f9d7_45bd_8dd1_57d79812916f.slice/crio-7a856324bc56e47e996cf26196a35488893a142dcecdaae8125a9b53b1f1250b WatchSource:0}: Error finding container 7a856324bc56e47e996cf26196a35488893a142dcecdaae8125a9b53b1f1250b: Status 404 returned error can't find the container with id 7a856324bc56e47e996cf26196a35488893a142dcecdaae8125a9b53b1f1250b
Sep 18 14:11:22.856240 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:22.856216    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7770-82482c"]
Sep 18 14:11:22.857514 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:11:22.857475    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5202ebe1_2a17_471b_8d63_deaf71e2fef0.slice/crio-9bb2a8dccc18d8b0e99a1a47964e1173901bb483ba5754ec9a38faff7f1138d9 WatchSource:0}: Error finding container 9bb2a8dccc18d8b0e99a1a47964e1173901bb483ba5754ec9a38faff7f1138d9: Status 404 returned error can't find the container with id 9bb2a8dccc18d8b0e99a1a47964e1173901bb483ba5754ec9a38faff7f1138d9
Sep 18 14:11:23.344317 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:23.344282    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7770-82482c" event={"ID":"5202ebe1-2a17-471b-8d63-deaf71e2fef0","Type":"ContainerStarted","Data":"b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a"}
Sep 18 14:11:23.344727 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:23.344328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7770-82482c" event={"ID":"5202ebe1-2a17-471b-8d63-deaf71e2fef0","Type":"ContainerStarted","Data":"9bb2a8dccc18d8b0e99a1a47964e1173901bb483ba5754ec9a38faff7f1138d9"}
Sep 18 14:11:23.345494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:23.345474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx" event={"ID":"dcaf20b7-f9d7-45bd-8dd1-57d79812916f","Type":"ContainerStarted","Data":"7a856324bc56e47e996cf26196a35488893a142dcecdaae8125a9b53b1f1250b"}
Sep 18 14:11:23.345568 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:23.345545    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7770-579034" podUID="ede154db-71c0-4c7b-ad9c-e722cc2284ab" containerName="test-curl-7770-579034" containerID="cri-o://daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373" gracePeriod=2
Sep 18 14:11:23.366922 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:23.366874    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7770-82482c" podStartSLOduration=1.3668597390000001 podStartE2EDuration="1.366859739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:23.363929731 +0000 UTC m=+771.520450013" watchObservedRunningTime="2026-09-18 14:11:23.366859739 +0000 UTC m=+771.523380053"
Sep 18 14:11:24.012907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.012863    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7770-82482c"]
Sep 18 14:11:24.019758 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.019729    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7770-82482c"]
Sep 18 14:11:24.354099 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.354052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx" event={"ID":"dcaf20b7-f9d7-45bd-8dd1-57d79812916f","Type":"ContainerStarted","Data":"b1689ca5711d03e871d6a080ac14e8db1c82a83035bd491e194f44ba52b5fb1c"}
Sep 18 14:11:24.354590 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.354172    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:11:24.354590 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:11:24.354352    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7770-82482c" container="test-curl-7770-82482c"
Sep 18 14:11:24.380057 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.379996    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx" podStartSLOduration=1.077792009 podStartE2EDuration="2.379978068s" totalImagesPullingTime="1.302186059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:24.378630158 +0000 UTC m=+772.535150438" watchObservedRunningTime="2026-09-18 14:11:24.379978068 +0000 UTC m=+772.536498377"
Sep 18 14:11:24.835329 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.835294    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"]
Sep 18 14:11:24.835674 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.835659    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5202ebe1-2a17-471b-8d63-deaf71e2fef0" containerName="test-curl-7770-82482c"
Sep 18 14:11:24.835723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.835676    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5202ebe1-2a17-471b-8d63-deaf71e2fef0" containerName="test-curl-7770-82482c"
Sep 18 14:11:24.835768 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.835747    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5202ebe1-2a17-471b-8d63-deaf71e2fef0" containerName="test-curl-7770-82482c"
Sep 18 14:11:24.838949 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.838925    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:24.841551 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.841528    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 14:11:24.841680 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.841663    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-ae7021-openshift-default-dockercfg-4rnb7\""
Sep 18 14:11:24.859644 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.859618    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"]
Sep 18 14:11:24.935211 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.935178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-data\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:24.935393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.935234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cvbq\" (UniqueName: \"kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-kube-api-access-2cvbq\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:24.935393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.935365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-envoy\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:24.935499 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.935409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-podinfo\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:24.935499 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.935439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istiod-ca-cert\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:24.935499 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.935477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-socket\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:24.935627 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.935542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-certs\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:24.935627 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.935579    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-token\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:24.935627 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:24.935618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-credential-socket\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.036517 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.036463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-data\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.036517 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.036506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cvbq\" (UniqueName: \"kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-kube-api-access-2cvbq\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.036775 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.036567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-envoy\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.036775 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.036599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-podinfo\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.036775 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.036624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istiod-ca-cert\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.036775 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.036660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-socket\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.036775 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.036690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-certs\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.036775 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.036727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-token\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.036775 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.036763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-credential-socket\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.037145 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.036959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-data\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.037344 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.037302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-credential-socket\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.037475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.037375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-certs\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.037475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.037415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-socket\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.037710 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.037691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istiod-ca-cert\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.039344 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.039320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-podinfo\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.039596 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.039579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-envoy\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.049224 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.049205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-token\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.049704 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.049683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cvbq\" (UniqueName: \"kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-kube-api-access-2cvbq\") pod \"e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") " pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.149674 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.149583    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:25.360058 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.360026    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7770-82482c" podUID="5202ebe1-2a17-471b-8d63-deaf71e2fef0" containerName="test-curl-7770-82482c" containerID="cri-o://b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a" gracePeriod=2
Sep 18 14:11:25.441235 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.441144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"]
Sep 18 14:11:25.442286 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:11:25.442251    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf82a9bd_e6e2_4cb7_ae7a_998baa8bf155.slice/crio-9a52c9f1351de6b2c1b98f7853b7bb5b73e0a4a68c01bd330a0c303e68e6eae9 WatchSource:0}: Error finding container 9a52c9f1351de6b2c1b98f7853b7bb5b73e0a4a68c01bd330a0c303e68e6eae9: Status 404 returned error can't find the container with id 9a52c9f1351de6b2c1b98f7853b7bb5b73e0a4a68c01bd330a0c303e68e6eae9
Sep 18 14:11:25.501182 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.501143    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-579034"
Sep 18 14:11:25.642913 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.642885    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5qr2\" (UniqueName: \"kubernetes.io/projected/ede154db-71c0-4c7b-ad9c-e722cc2284ab-kube-api-access-s5qr2\") pod \"ede154db-71c0-4c7b-ad9c-e722cc2284ab\" (UID: \"ede154db-71c0-4c7b-ad9c-e722cc2284ab\") "
Sep 18 14:11:25.645001 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.644982    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede154db-71c0-4c7b-ad9c-e722cc2284ab-kube-api-access-s5qr2" (OuterVolumeSpecName: "kube-api-access-s5qr2") pod "ede154db-71c0-4c7b-ad9c-e722cc2284ab" (UID: "ede154db-71c0-4c7b-ad9c-e722cc2284ab"). InnerVolumeSpecName "kube-api-access-s5qr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:11:25.743916 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:25.743840    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5qr2\" (UniqueName: \"kubernetes.io/projected/ede154db-71c0-4c7b-ad9c-e722cc2284ab-kube-api-access-s5qr2\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:11:26.293889 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:26.293768    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ede154db-71c0-4c7b-ad9c-e722cc2284ab" path="/var/lib/kubelet/pods/ede154db-71c0-4c7b-ad9c-e722cc2284ab/volumes"
Sep 18 14:11:26.368776 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:26.368724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" event={"ID":"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155","Type":"ContainerStarted","Data":"9a52c9f1351de6b2c1b98f7853b7bb5b73e0a4a68c01bd330a0c303e68e6eae9"}
Sep 18 14:11:26.370607 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:26.370539    2563 generic.go:356] "Generic (PLEG): container finished" podID="ede154db-71c0-4c7b-ad9c-e722cc2284ab" containerID="daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373" exitCode=137
Sep 18 14:11:26.370607 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:26.370603    2563 scope.go:117] "RemoveContainer" containerID="daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373"
Sep 18 14:11:26.370806 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:26.370579    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-579034"
Sep 18 14:11:26.381296 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:26.381279    2563 scope.go:117] "RemoveContainer" containerID="daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373"
Sep 18 14:11:26.381616 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:11:26.381590    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373\": container with ID starting with daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373 not found: ID does not exist" containerID="daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373"
Sep 18 14:11:26.381714 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:26.381628    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373"} err="failed to get container status \"daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373\": rpc error: code = NotFound desc = could not find container \"daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373\": container with ID starting with daf2533763c53deca346528bb49e88b3199d72d129d7b3d428de7eae72400373 not found: ID does not exist"
Sep 18 14:11:28.024886 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.024834    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:11:28.025273 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.024946    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:11:28.025273 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.024987    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:11:28.108941 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.108923    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-82482c"
Sep 18 14:11:28.267303 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.267215    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jvng\" (UniqueName: \"kubernetes.io/projected/5202ebe1-2a17-471b-8d63-deaf71e2fef0-kube-api-access-2jvng\") pod \"5202ebe1-2a17-471b-8d63-deaf71e2fef0\" (UID: \"5202ebe1-2a17-471b-8d63-deaf71e2fef0\") "
Sep 18 14:11:28.269394 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.269364    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5202ebe1-2a17-471b-8d63-deaf71e2fef0-kube-api-access-2jvng" (OuterVolumeSpecName: "kube-api-access-2jvng") pod "5202ebe1-2a17-471b-8d63-deaf71e2fef0" (UID: "5202ebe1-2a17-471b-8d63-deaf71e2fef0"). InnerVolumeSpecName "kube-api-access-2jvng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:11:28.289392 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.289356    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5202ebe1-2a17-471b-8d63-deaf71e2fef0" path="/var/lib/kubelet/pods/5202ebe1-2a17-471b-8d63-deaf71e2fef0/volumes"
Sep 18 14:11:28.368469 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.368433    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvng\" (UniqueName: \"kubernetes.io/projected/5202ebe1-2a17-471b-8d63-deaf71e2fef0-kube-api-access-2jvng\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:11:28.385963 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.385929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" event={"ID":"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155","Type":"ContainerStarted","Data":"d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa"}
Sep 18 14:11:28.387210 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.387185    2563 generic.go:356] "Generic (PLEG): container finished" podID="5202ebe1-2a17-471b-8d63-deaf71e2fef0" containerID="b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a" exitCode=137
Sep 18 14:11:28.387322 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.387225    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7770-82482c"
Sep 18 14:11:28.387322 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.387253    2563 scope.go:117] "RemoveContainer" containerID="b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a"
Sep 18 14:11:28.396961 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.396940    2563 scope.go:117] "RemoveContainer" containerID="b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a"
Sep 18 14:11:28.397294 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:11:28.397264    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a\": container with ID starting with b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a not found: ID does not exist" containerID="b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a"
Sep 18 14:11:28.397368 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.397307    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a"} err="failed to get container status \"b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a\": rpc error: code = NotFound desc = could not find container \"b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a\": container with ID starting with b8e8eac5b9460bd46e48c7dbb1afcf570663286060c2e26df4a2820405fde65a not found: ID does not exist"
Sep 18 14:11:28.414986 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:28.414919    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" podStartSLOduration=1.83546757 podStartE2EDuration="4.414906402s" totalImagesPullingTime="2.579438832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:28.41325809 +0000 UTC m=+776.569778372" watchObservedRunningTime="2026-09-18 14:11:28.414906402 +0000 UTC m=+776.571426683"
Sep 18 14:11:29.150116 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:29.150079    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:29.156315 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:29.156285    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:29.394427 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:29.394390    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:29.395407 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:29.395383    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:31.930499 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:31.930460    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"]
Sep 18 14:11:31.935404 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:31.935381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:31.938054 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:31.938014    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5nh5f\""
Sep 18 14:11:31.938209 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:31.938073    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-eeb0a3f4-serving-cert\""
Sep 18 14:11:31.938289 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:31.938197    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 14:11:31.938965 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:31.938929    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 14:11:31.939207 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:31.939034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 14:11:31.946276 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:31.946252    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"]
Sep 18 14:11:32.103197 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.103158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-maas-api-tls\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.103392 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.103205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-metrics-certs\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.103392 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.103361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f273ff05-db26-4335-8f95-04284b083607-openshift-service-ca\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.103501 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.103403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt6vr\" (UniqueName: \"kubernetes.io/projected/f273ff05-db26-4335-8f95-04284b083607-kube-api-access-xt6vr\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.204818 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.204728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f273ff05-db26-4335-8f95-04284b083607-openshift-service-ca\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.204818 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.204779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt6vr\" (UniqueName: \"kubernetes.io/projected/f273ff05-db26-4335-8f95-04284b083607-kube-api-access-xt6vr\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.205056 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.204840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-maas-api-tls\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.205056 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.204872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-metrics-certs\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.207415 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.207387    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-eeb0a3f4-serving-cert\""
Sep 18 14:11:32.207539 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.207429    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 14:11:32.207539 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.207443    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 14:11:32.213122 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.213105    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 14:11:32.215414 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.215394    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f273ff05-db26-4335-8f95-04284b083607-openshift-service-ca\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.215496 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:11:32.215463    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-serving-cert: secret "maas-api-e2e-ait-eeb0a3f4-serving-cert" not found
Sep 18 14:11:32.215537 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:11:32.215525    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-maas-api-tls podName:f273ff05-db26-4335-8f95-04284b083607 nodeName:}" failed. No retries permitted until 2026-09-18 14:11:32.715507094 +0000 UTC m=+780.872027368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-maas-api-tls") pod "maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s" (UID: "f273ff05-db26-4335-8f95-04284b083607") : secret "maas-api-e2e-ait-eeb0a3f4-serving-cert" not found
Sep 18 14:11:32.217647 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.217628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-metrics-certs\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.223721 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.223703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt6vr\" (UniqueName: \"kubernetes.io/projected/f273ff05-db26-4335-8f95-04284b083607-kube-api-access-xt6vr\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.809713 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.809546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-maas-api-tls\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.812296 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.812266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-maas-api-tls\") pod \"maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.850287 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.850266    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5nh5f\""
Sep 18 14:11:32.858267 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.858247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:32.985968 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.985940    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"]
Sep 18 14:11:32.990413 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.990386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:11:32.992961 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.992931    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-eeb0a3f4-dockercfg-qqwqr\""
Sep 18 14:11:32.993209 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.993195    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-eeb0a3f4\""
Sep 18 14:11:32.999902 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:32.999855    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"]
Sep 18 14:11:33.112156 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:33.112059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl8k7\" (UniqueName: \"kubernetes.io/projected/077ff23c-5df0-4623-a4c5-5899013850a2-kube-api-access-bl8k7\") pod \"payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l\" (UID: \"077ff23c-5df0-4623-a4c5-5899013850a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:11:33.112156 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:33.112101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077ff23c-5df0-4623-a4c5-5899013850a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l\" (UID: \"077ff23c-5df0-4623-a4c5-5899013850a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:11:33.143314 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:33.143290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"]
Sep 18 14:11:33.145331 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:11:33.145299    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf273ff05_db26_4335_8f95_04284b083607.slice/crio-dc3fa3dc194fdd7a8c70c0c4b39bae92599dd5b6a0c70faf98eaba1861cefea2 WatchSource:0}: Error finding container dc3fa3dc194fdd7a8c70c0c4b39bae92599dd5b6a0c70faf98eaba1861cefea2: Status 404 returned error can't find the container with id dc3fa3dc194fdd7a8c70c0c4b39bae92599dd5b6a0c70faf98eaba1861cefea2
Sep 18 14:11:33.212631 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:33.212604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl8k7\" (UniqueName: \"kubernetes.io/projected/077ff23c-5df0-4623-a4c5-5899013850a2-kube-api-access-bl8k7\") pod \"payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l\" (UID: \"077ff23c-5df0-4623-a4c5-5899013850a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:11:33.212727 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:33.212643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077ff23c-5df0-4623-a4c5-5899013850a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l\" (UID: \"077ff23c-5df0-4623-a4c5-5899013850a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:11:33.213291 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:33.213271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077ff23c-5df0-4623-a4c5-5899013850a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l\" (UID: \"077ff23c-5df0-4623-a4c5-5899013850a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:11:33.220762 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:33.220744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl8k7\" (UniqueName: \"kubernetes.io/projected/077ff23c-5df0-4623-a4c5-5899013850a2-kube-api-access-bl8k7\") pod \"payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l\" (UID: \"077ff23c-5df0-4623-a4c5-5899013850a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:11:33.301734 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:33.301713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:11:33.416324 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:33.416252    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s" event={"ID":"f273ff05-db26-4335-8f95-04284b083607","Type":"ContainerStarted","Data":"dc3fa3dc194fdd7a8c70c0c4b39bae92599dd5b6a0c70faf98eaba1861cefea2"}
Sep 18 14:11:33.575174 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:33.575143    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"]
Sep 18 14:11:33.576239 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:11:33.576210    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod077ff23c_5df0_4623_a4c5_5899013850a2.slice/crio-f88c7d3faef4db0941e61eb13c79296c8ff41d0a0318af54087810c1c4ba458d WatchSource:0}: Error finding container f88c7d3faef4db0941e61eb13c79296c8ff41d0a0318af54087810c1c4ba458d: Status 404 returned error can't find the container with id f88c7d3faef4db0941e61eb13c79296c8ff41d0a0318af54087810c1c4ba458d
Sep 18 14:11:34.425670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:34.425635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l" event={"ID":"077ff23c-5df0-4623-a4c5-5899013850a2","Type":"ContainerStarted","Data":"1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3"}
Sep 18 14:11:34.426117 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:34.425699    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l" event={"ID":"077ff23c-5df0-4623-a4c5-5899013850a2","Type":"ContainerStarted","Data":"f88c7d3faef4db0941e61eb13c79296c8ff41d0a0318af54087810c1c4ba458d"}
Sep 18 14:11:34.426117 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:34.425860    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:11:34.444943 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:34.444565    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l" podStartSLOduration=2.444548136 podStartE2EDuration="2.444548136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:34.444084006 +0000 UTC m=+782.600604287" watchObservedRunningTime="2026-09-18 14:11:34.444548136 +0000 UTC m=+782.601068414"
Sep 18 14:11:35.362450 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:35.362416    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:11:36.222550 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.222508    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"]
Sep 18 14:11:36.226341 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.226317    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:11:36.228684 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.228661    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ae7021\""
Sep 18 14:11:36.228778 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.228754    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ae7021-dockercfg-jngmp\""
Sep 18 14:11:36.239547 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.239520    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa8de16-1e8c-4f2b-af20-5111aee62152-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj\" (UID: \"2fa8de16-1e8c-4f2b-af20-5111aee62152\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:11:36.239679 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.239618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztn7\" (UniqueName: \"kubernetes.io/projected/2fa8de16-1e8c-4f2b-af20-5111aee62152-kube-api-access-6ztn7\") pod \"payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj\" (UID: \"2fa8de16-1e8c-4f2b-af20-5111aee62152\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:11:36.242549 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.242527    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"]
Sep 18 14:11:36.252407 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.252382    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"]
Sep 18 14:11:36.259008 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.258986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:11:36.281396 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.281374    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"]
Sep 18 14:11:36.340373 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.340342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa8de16-1e8c-4f2b-af20-5111aee62152-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj\" (UID: \"2fa8de16-1e8c-4f2b-af20-5111aee62152\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:11:36.340532 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.340391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzm8\" (UniqueName: \"kubernetes.io/projected/68a3f9ec-7666-43ec-9d35-3935637e885b-kube-api-access-szzm8\") pod \"payload-processing-e2e-autores-ae7021-648bccd949-qvzld\" (UID: \"68a3f9ec-7666-43ec-9d35-3935637e885b\") " pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:11:36.340532 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.340438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztn7\" (UniqueName: \"kubernetes.io/projected/2fa8de16-1e8c-4f2b-af20-5111aee62152-kube-api-access-6ztn7\") pod \"payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj\" (UID: \"2fa8de16-1e8c-4f2b-af20-5111aee62152\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:11:36.340532 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.340494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a3f9ec-7666-43ec-9d35-3935637e885b-plugins-config-volume\") pod \"payload-processing-e2e-autores-ae7021-648bccd949-qvzld\" (UID: \"68a3f9ec-7666-43ec-9d35-3935637e885b\") " pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:11:36.341141 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.341101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa8de16-1e8c-4f2b-af20-5111aee62152-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj\" (UID: \"2fa8de16-1e8c-4f2b-af20-5111aee62152\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:11:36.349156 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.349118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztn7\" (UniqueName: \"kubernetes.io/projected/2fa8de16-1e8c-4f2b-af20-5111aee62152-kube-api-access-6ztn7\") pod \"payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj\" (UID: \"2fa8de16-1e8c-4f2b-af20-5111aee62152\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:11:36.440557 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.440526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s" event={"ID":"f273ff05-db26-4335-8f95-04284b083607","Type":"ContainerStarted","Data":"2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51"}
Sep 18 14:11:36.440701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.440634    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:36.440947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.440924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzm8\" (UniqueName: \"kubernetes.io/projected/68a3f9ec-7666-43ec-9d35-3935637e885b-kube-api-access-szzm8\") pod \"payload-processing-e2e-autores-ae7021-648bccd949-qvzld\" (UID: \"68a3f9ec-7666-43ec-9d35-3935637e885b\") " pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:11:36.440993 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.440980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a3f9ec-7666-43ec-9d35-3935637e885b-plugins-config-volume\") pod \"payload-processing-e2e-autores-ae7021-648bccd949-qvzld\" (UID: \"68a3f9ec-7666-43ec-9d35-3935637e885b\") " pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:11:36.441704 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.441675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a3f9ec-7666-43ec-9d35-3935637e885b-plugins-config-volume\") pod \"payload-processing-e2e-autores-ae7021-648bccd949-qvzld\" (UID: \"68a3f9ec-7666-43ec-9d35-3935637e885b\") " pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:11:36.445980 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.445955    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:11:36.454992 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.454965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzm8\" (UniqueName: \"kubernetes.io/projected/68a3f9ec-7666-43ec-9d35-3935637e885b-kube-api-access-szzm8\") pod \"payload-processing-e2e-autores-ae7021-648bccd949-qvzld\" (UID: \"68a3f9ec-7666-43ec-9d35-3935637e885b\") " pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:11:36.480291 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.480200    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s" podStartSLOduration=3.246772022 podStartE2EDuration="5.480183413s" totalImagesPullingTime="2.233411391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:36.478589899 +0000 UTC m=+784.635110182" watchObservedRunningTime="2026-09-18 14:11:36.480183413 +0000 UTC m=+784.636703694"
Sep 18 14:11:36.536400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.536370    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:11:36.569525 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.569495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:11:36.826187 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.826157    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"]
Sep 18 14:11:36.826774 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:11:36.826747    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa8de16_1e8c_4f2b_af20_5111aee62152.slice/crio-4934b0a9e00eada24c88bca427b8d8e90038c414065ea1b47cacf2e7fd693d47 WatchSource:0}: Error finding container 4934b0a9e00eada24c88bca427b8d8e90038c414065ea1b47cacf2e7fd693d47: Status 404 returned error can't find the container with id 4934b0a9e00eada24c88bca427b8d8e90038c414065ea1b47cacf2e7fd693d47
Sep 18 14:11:36.865797 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:36.865769    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"]
Sep 18 14:11:36.867661 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:11:36.867634    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a3f9ec_7666_43ec_9d35_3935637e885b.slice/crio-7a873b208de3639c6bd27f4208f28d89651bf1340545142b1d539e138992ff8c WatchSource:0}: Error finding container 7a873b208de3639c6bd27f4208f28d89651bf1340545142b1d539e138992ff8c: Status 404 returned error can't find the container with id 7a873b208de3639c6bd27f4208f28d89651bf1340545142b1d539e138992ff8c
Sep 18 14:11:37.449791 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:37.449756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj" event={"ID":"2fa8de16-1e8c-4f2b-af20-5111aee62152","Type":"ContainerStarted","Data":"5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2"}
Sep 18 14:11:37.450277 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:37.449817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj" event={"ID":"2fa8de16-1e8c-4f2b-af20-5111aee62152","Type":"ContainerStarted","Data":"4934b0a9e00eada24c88bca427b8d8e90038c414065ea1b47cacf2e7fd693d47"}
Sep 18 14:11:37.450277 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:37.449851    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:11:37.451722 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:37.451688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld" event={"ID":"68a3f9ec-7666-43ec-9d35-3935637e885b","Type":"ContainerStarted","Data":"a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7"}
Sep 18 14:11:37.451858 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:37.451745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld" event={"ID":"68a3f9ec-7666-43ec-9d35-3935637e885b","Type":"ContainerStarted","Data":"7a873b208de3639c6bd27f4208f28d89651bf1340545142b1d539e138992ff8c"}
Sep 18 14:11:37.487335 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:37.487285    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj" podStartSLOduration=1.4872707250000001 podStartE2EDuration="1.487270725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:37.485006183 +0000 UTC m=+785.641526464" watchObservedRunningTime="2026-09-18 14:11:37.487270725 +0000 UTC m=+785.643791006"
Sep 18 14:11:37.509889 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:37.509841    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld" podStartSLOduration=1.509827156 podStartE2EDuration="1.509827156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:37.507584423 +0000 UTC m=+785.664104706" watchObservedRunningTime="2026-09-18 14:11:37.509827156 +0000 UTC m=+785.666347437"
Sep 18 14:11:38.457798 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:38.457762    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:11:42.414384 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.414344    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"]
Sep 18 14:11:42.418823 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.418799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.421487 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.421463    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ee8a08-openshift-default-dockercfg-qxgpv\""
Sep 18 14:11:42.435205 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.435179    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"]
Sep 18 14:11:42.458907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.458878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:11:42.495175 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.495144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-data\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.495344 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.495207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-socket\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.495344 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.495309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-envoy\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.495470 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.495377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-podinfo\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.495470 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.495428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5b84\" (UniqueName: \"kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-kube-api-access-k5b84\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.495581 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.495522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-token\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.495661 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.495637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-credential-socket\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.495713 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.495676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-certs\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.495761 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.495713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istiod-ca-cert\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.596762 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.596725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-socket\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.596949 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.596789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-envoy\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.596949 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.596832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-podinfo\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.596949 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.596867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5b84\" (UniqueName: \"kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-kube-api-access-k5b84\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.596949 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.596910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-token\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.596949 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.596945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-credential-socket\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.597248 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.596971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-certs\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.597248 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.597005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istiod-ca-cert\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.597248 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.597056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-data\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.597824 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.597797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-socket\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.597981 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.597831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-credential-socket\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.597981 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.597852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-certs\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.598119 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.597902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istiod-ca-cert\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.598119 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.597919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-data\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.599528 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.599504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-podinfo\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.599703 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.599688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-envoy\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.609622 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.609589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-token\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.627078 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.627055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5b84\" (UniqueName: \"kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-kube-api-access-k5b84\") pod \"e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") " pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:42.731870 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:42.731840    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:43.010435 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:43.010403    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"]
Sep 18 14:11:43.011822 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:11:43.011794    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ba982a_4a96_4d3c_b8b4_d832bfc2e3fb.slice/crio-20b92ee37730f4b46e1ef4a75ea49106b015505ebabf0e896ded7abecbf23e8e WatchSource:0}: Error finding container 20b92ee37730f4b46e1ef4a75ea49106b015505ebabf0e896ded7abecbf23e8e: Status 404 returned error can't find the container with id 20b92ee37730f4b46e1ef4a75ea49106b015505ebabf0e896ded7abecbf23e8e
Sep 18 14:11:43.014269 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:43.014233    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:11:43.014349 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:43.014304    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:11:43.014349 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:43.014337    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:11:43.487620 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:43.487586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" event={"ID":"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb","Type":"ContainerStarted","Data":"0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d"}
Sep 18 14:11:43.488003 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:43.487640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" event={"ID":"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb","Type":"ContainerStarted","Data":"20b92ee37730f4b46e1ef4a75ea49106b015505ebabf0e896ded7abecbf23e8e"}
Sep 18 14:11:43.510548 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:43.510489    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" podStartSLOduration=1.510472198 podStartE2EDuration="1.510472198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:43.507743114 +0000 UTC m=+791.664263397" watchObservedRunningTime="2026-09-18 14:11:43.510472198 +0000 UTC m=+791.666992478"
Sep 18 14:11:43.732486 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:43.732450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:44.737939 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:44.737898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:45.435575 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:45.435538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:11:45.509552 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:45.509511    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:45.514674 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:45.514625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:11:48.459759 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:48.459725    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:11:49.465799 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:49.465773    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:11:51.679068 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.679033    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"]
Sep 18 14:11:51.686282 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.686260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:11:51.690159 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.690139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ee8a08\""
Sep 18 14:11:51.719407 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.719363    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"]
Sep 18 14:11:51.785384 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.785346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfmqc\" (UniqueName: \"kubernetes.io/projected/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-kube-api-access-mfmqc\") pod \"payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6\" (UID: \"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:11:51.785594 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.785519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6\" (UID: \"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:11:51.886913 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.886877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6\" (UID: \"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:11:51.887097 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.886930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfmqc\" (UniqueName: \"kubernetes.io/projected/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-kube-api-access-mfmqc\") pod \"payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6\" (UID: \"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:11:51.887870 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.887845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6\" (UID: \"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:11:51.907814 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.907776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfmqc\" (UniqueName: \"kubernetes.io/projected/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-kube-api-access-mfmqc\") pod \"payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6\" (UID: \"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:11:51.970929 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.970832    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"]
Sep 18 14:11:51.973403 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.973380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:11:51.985856 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:11:51.985825    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="2387a594-7667-4484-9eb6-731d02d12e8c" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:11:51.986159 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:11:51.986122    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-inf-b-ee8a08-dockercfg-6qxwq\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ee8a08-dockercfg-6qxwq\"" type="*v1.Secret"
Sep 18 14:11:51.997059 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.997021    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:11:51.999414 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:51.999381    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"]
Sep 18 14:11:52.088790 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:52.088750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dqb9\" (UniqueName: \"kubernetes.io/projected/2387a594-7667-4484-9eb6-731d02d12e8c-kube-api-access-2dqb9\") pod \"payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f\" (UID: \"2387a594-7667-4484-9eb6-731d02d12e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:11:52.089024 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:52.088817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2387a594-7667-4484-9eb6-731d02d12e8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f\" (UID: \"2387a594-7667-4484-9eb6-731d02d12e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:11:52.190186 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:52.190111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dqb9\" (UniqueName: \"kubernetes.io/projected/2387a594-7667-4484-9eb6-731d02d12e8c-kube-api-access-2dqb9\") pod \"payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f\" (UID: \"2387a594-7667-4484-9eb6-731d02d12e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:11:52.190386 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:52.190212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2387a594-7667-4484-9eb6-731d02d12e8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f\" (UID: \"2387a594-7667-4484-9eb6-731d02d12e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:11:52.190975 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:52.190950    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2387a594-7667-4484-9eb6-731d02d12e8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f\" (UID: \"2387a594-7667-4484-9eb6-731d02d12e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:11:52.228289 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:52.228202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dqb9\" (UniqueName: \"kubernetes.io/projected/2387a594-7667-4484-9eb6-731d02d12e8c-kube-api-access-2dqb9\") pod \"payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f\" (UID: \"2387a594-7667-4484-9eb6-731d02d12e8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:11:52.309364 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:52.309241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"]
Sep 18 14:11:52.554855 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:52.554757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6" event={"ID":"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8","Type":"ContainerStarted","Data":"cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823"}
Sep 18 14:11:52.554855 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:52.554812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6" event={"ID":"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8","Type":"ContainerStarted","Data":"42b7aa0607e4b60726ef31bdd61aeae48420b7cf766331a1bc2626fa66e6b15a"}
Sep 18 14:11:52.554855 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:52.554844    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:11:52.577467 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:52.577415    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6" podStartSLOduration=1.577400045 podStartE2EDuration="1.577400045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:52.575763732 +0000 UTC m=+800.732284014" watchObservedRunningTime="2026-09-18 14:11:52.577400045 +0000 UTC m=+800.733920326"
Sep 18 14:11:53.285068 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:53.285027    2563 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f" secret="" err="failed to sync secret cache: timed out waiting for the condition"
Sep 18 14:11:53.285471 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:53.285103    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:11:53.379684 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:53.379650    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ee8a08-dockercfg-6qxwq\""
Sep 18 14:11:53.605804 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:53.605770    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"]
Sep 18 14:11:53.613925 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:11:53.613887    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2387a594_7667_4484_9eb6_731d02d12e8c.slice/crio-df310eecb6a85bfc32b8ce2e01709227d68771ef09814114b4bded1bb0d1db22 WatchSource:0}: Error finding container df310eecb6a85bfc32b8ce2e01709227d68771ef09814114b4bded1bb0d1db22: Status 404 returned error can't find the container with id df310eecb6a85bfc32b8ce2e01709227d68771ef09814114b4bded1bb0d1db22
Sep 18 14:11:54.569978 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:54.569940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f" event={"ID":"2387a594-7667-4484-9eb6-731d02d12e8c","Type":"ContainerStarted","Data":"858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b"}
Sep 18 14:11:54.570369 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:54.569999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f" event={"ID":"2387a594-7667-4484-9eb6-731d02d12e8c","Type":"ContainerStarted","Data":"df310eecb6a85bfc32b8ce2e01709227d68771ef09814114b4bded1bb0d1db22"}
Sep 18 14:11:54.570369 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:54.570025    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:11:54.592972 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:54.592917    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f" podStartSLOduration=3.592900509 podStartE2EDuration="3.592900509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:11:54.591497063 +0000 UTC m=+802.748017348" watchObservedRunningTime="2026-09-18 14:11:54.592900509 +0000 UTC m=+802.749420790"
Sep 18 14:11:57.591959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:57.591929    2563 generic.go:356] "Generic (PLEG): container finished" podID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerID="d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa" exitCode=0
Sep 18 14:11:57.592545 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:57.592001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" event={"ID":"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155","Type":"ContainerDied","Data":"d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa"}
Sep 18 14:11:57.592545 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:57.592422    2563 scope.go:117] "RemoveContainer" containerID="d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa"
Sep 18 14:11:57.593541 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:57.593511    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:11:57.593627 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:57.593578    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:11:57.593627 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:57.593608    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:11:58.600220 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:58.600188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" event={"ID":"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155","Type":"ContainerStarted","Data":"5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd"}
Sep 18 14:11:59.134389 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:59.134345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:11:59.149875 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:59.149833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:11:59.150947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:59.150924    2563 patch_prober.go:28] interesting pod/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 18 14:11:59.151016 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:59.150975    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 18 14:11:59.394752 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:11:59.394649    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:12:00.150754 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:00.150719    2563 patch_prober.go:28] interesting pod/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 18 14:12:00.151157 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:00.150775    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 18 14:12:01.150258 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:01.150218    2563 patch_prober.go:28] interesting pod/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 18 14:12:01.150555 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:01.150522    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 18 14:12:02.151103 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:02.151067    2563 patch_prober.go:28] interesting pod/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 18 14:12:02.151749 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:02.151715    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 18 14:12:03.150896 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.150857    2563 patch_prober.go:28] interesting pod/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 18 14:12:03.151214 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.151184    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 18 14:12:03.564925 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.564891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:12:03.747167 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.744968    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"]
Sep 18 14:12:03.759678 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.759648    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"]
Sep 18 14:12:03.759849 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.759773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.763806 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.763771    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4d9222\"/\"default-dockercfg-st5nr\""
Sep 18 14:12:03.768122 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.768082    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4d9222\"/\"kube-root-ca.crt\""
Sep 18 14:12:03.768348 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.768327    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4d9222\"/\"openshift-service-ca.crt\""
Sep 18 14:12:03.768713 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.768539    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4d9222\"/\"simulator-w0-4d9222-kserve-self-signed-certs\""
Sep 18 14:12:03.801572 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.801545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-model-cache\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.801735 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.801648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrbk8\" (UniqueName: \"kubernetes.io/projected/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-kube-api-access-rrbk8\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.801735 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.801694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-dshm\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.801865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.801744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tmp-dir\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.801865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.801781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tls-certs\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.801865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.801813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-home\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.902976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.902886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tmp-dir\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.902976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.902931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tls-certs\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.902976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.902952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-home\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.903309 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.903080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-model-cache\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.903309 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.903215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrbk8\" (UniqueName: \"kubernetes.io/projected/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-kube-api-access-rrbk8\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.903309 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.903258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-dshm\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.903465 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.903408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tmp-dir\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.903465 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.903436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-home\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.906156 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.903709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-model-cache\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.907375 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.907352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-dshm\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.912743 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.912712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tls-certs\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:03.920449 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:03.920418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrbk8\" (UniqueName: \"kubernetes.io/projected/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-kube-api-access-rrbk8\") pod \"simulator-w0-4d9222-kserve-78cb7f4f88-29hw6\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") " pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:04.071352 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:04.071311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:04.179317 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:04.179193    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:12:04.181505 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:04.181471    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:12:04.378733 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:04.378702    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"]
Sep 18 14:12:04.378874 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:12:04.378746    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24aa8ad7_fee6_458b_bbb6_a6d48d697bc0.slice/crio-cc5bc010a371d392dd74488c76d27087b8f1db36ef265b9d342f4126b0c6fb0b WatchSource:0}: Error finding container cc5bc010a371d392dd74488c76d27087b8f1db36ef265b9d342f4126b0c6fb0b: Status 404 returned error can't find the container with id cc5bc010a371d392dd74488c76d27087b8f1db36ef265b9d342f4126b0c6fb0b
Sep 18 14:12:04.654535 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:04.654501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6" event={"ID":"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0","Type":"ContainerStarted","Data":"65b4fe15951eda56bb485cd081b9cec197452528cf8c31d77d78eeae62b23b89"}
Sep 18 14:12:04.654707 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:04.654547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6" event={"ID":"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0","Type":"ContainerStarted","Data":"cc5bc010a371d392dd74488c76d27087b8f1db36ef265b9d342f4126b0c6fb0b"}
Sep 18 14:12:04.678670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:04.678624    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6" podStartSLOduration=1.6786080939999999 podStartE2EDuration="1.678608094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:12:04.676426663 +0000 UTC m=+812.832946945" watchObservedRunningTime="2026-09-18 14:12:04.678608094 +0000 UTC m=+812.835128374"
Sep 18 14:12:05.579757 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:05.579715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:12:05.975174 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:05.975140    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"]
Sep 18 14:12:05.975368 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:05.975345    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s" podUID="f273ff05-db26-4335-8f95-04284b083607" containerName="maas-api" containerID="cri-o://2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51" gracePeriod=30
Sep 18 14:12:06.070308 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.070274    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"]
Sep 18 14:12:06.070591 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.070544    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l" podUID="077ff23c-5df0-4623-a4c5-5899013850a2" containerName="payload-pre-processing" containerID="cri-o://1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3" gracePeriod=30
Sep 18 14:12:06.247663 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.247631    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:12:06.333521 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.333497    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:12:06.349105 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.349076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-maas-api-tls\") pod \"f273ff05-db26-4335-8f95-04284b083607\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") "
Sep 18 14:12:06.349253 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.349119    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt6vr\" (UniqueName: \"kubernetes.io/projected/f273ff05-db26-4335-8f95-04284b083607-kube-api-access-xt6vr\") pod \"f273ff05-db26-4335-8f95-04284b083607\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") "
Sep 18 14:12:06.349253 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.349170    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-metrics-certs\") pod \"f273ff05-db26-4335-8f95-04284b083607\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") "
Sep 18 14:12:06.349253 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.349246    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f273ff05-db26-4335-8f95-04284b083607-openshift-service-ca\") pod \"f273ff05-db26-4335-8f95-04284b083607\" (UID: \"f273ff05-db26-4335-8f95-04284b083607\") "
Sep 18 14:12:06.349666 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.349641    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f273ff05-db26-4335-8f95-04284b083607-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f273ff05-db26-4335-8f95-04284b083607" (UID: "f273ff05-db26-4335-8f95-04284b083607"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:06.351313 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.351286    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f273ff05-db26-4335-8f95-04284b083607" (UID: "f273ff05-db26-4335-8f95-04284b083607"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:06.351400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.351376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f273ff05-db26-4335-8f95-04284b083607-kube-api-access-xt6vr" (OuterVolumeSpecName: "kube-api-access-xt6vr") pod "f273ff05-db26-4335-8f95-04284b083607" (UID: "f273ff05-db26-4335-8f95-04284b083607"). InnerVolumeSpecName "kube-api-access-xt6vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:06.351524 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.351504    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f273ff05-db26-4335-8f95-04284b083607" (UID: "f273ff05-db26-4335-8f95-04284b083607"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:12:06.449808 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.449780    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077ff23c-5df0-4623-a4c5-5899013850a2-plugins-config-volume\") pod \"077ff23c-5df0-4623-a4c5-5899013850a2\" (UID: \"077ff23c-5df0-4623-a4c5-5899013850a2\") "
Sep 18 14:12:06.449920 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.449866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl8k7\" (UniqueName: \"kubernetes.io/projected/077ff23c-5df0-4623-a4c5-5899013850a2-kube-api-access-bl8k7\") pod \"077ff23c-5df0-4623-a4c5-5899013850a2\" (UID: \"077ff23c-5df0-4623-a4c5-5899013850a2\") "
Sep 18 14:12:06.450107 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.450085    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077ff23c-5df0-4623-a4c5-5899013850a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "077ff23c-5df0-4623-a4c5-5899013850a2" (UID: "077ff23c-5df0-4623-a4c5-5899013850a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:06.450214 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.450109    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-metrics-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.450214 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.450157    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f273ff05-db26-4335-8f95-04284b083607-openshift-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.450214 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.450174    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f273ff05-db26-4335-8f95-04284b083607-maas-api-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.450214 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.450188    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt6vr\" (UniqueName: \"kubernetes.io/projected/f273ff05-db26-4335-8f95-04284b083607-kube-api-access-xt6vr\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.451915 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.451896    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077ff23c-5df0-4623-a4c5-5899013850a2-kube-api-access-bl8k7" (OuterVolumeSpecName: "kube-api-access-bl8k7") pod "077ff23c-5df0-4623-a4c5-5899013850a2" (UID: "077ff23c-5df0-4623-a4c5-5899013850a2"). InnerVolumeSpecName "kube-api-access-bl8k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:12:06.550873 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.550830    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/077ff23c-5df0-4623-a4c5-5899013850a2-plugins-config-volume\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.550873 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.550868    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl8k7\" (UniqueName: \"kubernetes.io/projected/077ff23c-5df0-4623-a4c5-5899013850a2-kube-api-access-bl8k7\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:12:06.670117 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.670075    2563 generic.go:356] "Generic (PLEG): container finished" podID="f273ff05-db26-4335-8f95-04284b083607" containerID="2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51" exitCode=0
Sep 18 14:12:06.670628 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.670117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s" event={"ID":"f273ff05-db26-4335-8f95-04284b083607","Type":"ContainerDied","Data":"2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51"}
Sep 18 14:12:06.670628 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.670142    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"
Sep 18 14:12:06.670628 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.670190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s" event={"ID":"f273ff05-db26-4335-8f95-04284b083607","Type":"ContainerDied","Data":"dc3fa3dc194fdd7a8c70c0c4b39bae92599dd5b6a0c70faf98eaba1861cefea2"}
Sep 18 14:12:06.670628 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.670222    2563 scope.go:117] "RemoveContainer" containerID="2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51"
Sep 18 14:12:06.672286 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.671985    2563 generic.go:356] "Generic (PLEG): container finished" podID="077ff23c-5df0-4623-a4c5-5899013850a2" containerID="1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3" exitCode=0
Sep 18 14:12:06.672286 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.672040    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"
Sep 18 14:12:06.672286 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.672174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l" event={"ID":"077ff23c-5df0-4623-a4c5-5899013850a2","Type":"ContainerDied","Data":"1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3"}
Sep 18 14:12:06.672286 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.672217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l" event={"ID":"077ff23c-5df0-4623-a4c5-5899013850a2","Type":"ContainerDied","Data":"f88c7d3faef4db0941e61eb13c79296c8ff41d0a0318af54087810c1c4ba458d"}
Sep 18 14:12:06.682369 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.682354    2563 scope.go:117] "RemoveContainer" containerID="2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51"
Sep 18 14:12:06.682659 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:12:06.682636    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51\": container with ID starting with 2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51 not found: ID does not exist" containerID="2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51"
Sep 18 14:12:06.682726 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.682669    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51"} err="failed to get container status \"2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51\": rpc error: code = NotFound desc = could not find container \"2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51\": container with ID starting with 2ced7c8acebd4fcb15896797ef2f1c3ffaeb565de68a31ad19d52f63bb802b51 not found: ID does not exist"
Sep 18 14:12:06.682726 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.682702    2563 scope.go:117] "RemoveContainer" containerID="1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3"
Sep 18 14:12:06.694361 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.694345    2563 scope.go:117] "RemoveContainer" containerID="1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3"
Sep 18 14:12:06.694608 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:12:06.694592    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3\": container with ID starting with 1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3 not found: ID does not exist" containerID="1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3"
Sep 18 14:12:06.694676 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.694619    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3"} err="failed to get container status \"1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3\": rpc error: code = NotFound desc = could not find container \"1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3\": container with ID starting with 1680201cce80ce5fcbb05530947dd7485e6f07fe5005c131421d88dac8e7b6f3 not found: ID does not exist"
Sep 18 14:12:06.711291 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.711230    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"]
Sep 18 14:12:06.723452 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.723424    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eeb0a3f4-6df86bf4f8-vzl5l"]
Sep 18 14:12:06.774694 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.774608    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"]
Sep 18 14:12:06.778881 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:06.778854    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-eeb0a3f4-6947d8b97b-m5c8s"]
Sep 18 14:12:08.289105 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:08.289067    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="077ff23c-5df0-4623-a4c5-5899013850a2" path="/var/lib/kubelet/pods/077ff23c-5df0-4623-a4c5-5899013850a2/volumes"
Sep 18 14:12:08.289545 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:08.289529    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f273ff05-db26-4335-8f95-04284b083607" path="/var/lib/kubelet/pods/f273ff05-db26-4335-8f95-04284b083607/volumes"
Sep 18 14:12:09.693579 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:09.693548    2563 generic.go:356] "Generic (PLEG): container finished" podID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerID="0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d" exitCode=0
Sep 18 14:12:09.693964 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:09.693597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" event={"ID":"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb","Type":"ContainerDied","Data":"0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d"}
Sep 18 14:12:09.693964 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:09.693936    2563 scope.go:117] "RemoveContainer" containerID="0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d"
Sep 18 14:12:09.695057 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:09.695026    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:12:09.695168 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:09.695106    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:12:09.695237 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:09.695173    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:12:10.701333 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:10.701296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" event={"ID":"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb","Type":"ContainerStarted","Data":"ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c"}
Sep 18 14:12:10.732540 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:10.732501    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:12:10.732725 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:10.732705    2563 patch_prober.go:28] interesting pod/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 18 14:12:10.732774 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:10.732752    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 18 14:12:11.732520 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:11.732483    2563 patch_prober.go:28] interesting pod/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 18 14:12:11.732967 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:11.732538    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 18 14:12:12.733036 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:12.733006    2563 patch_prober.go:28] interesting pod/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 18 14:12:12.733433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:12.733054    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 18 14:12:13.732738 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:13.732700    2563 patch_prober.go:28] interesting pod/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 18 14:12:13.732978 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:13.732763    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 18 14:12:13.915643 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:13.915602    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:12:14.071949 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:14.071871    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:14.071949 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:14.071912    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:14.084503 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:14.084476    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:14.739097 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:14.739052    2563 patch_prober.go:28] interesting pod/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 18 14:12:14.740167 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:14.740113    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 18 14:12:14.762400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:14.762377    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:12:15.511248 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:15.511215    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:12:15.822377 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:15.822280    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:12:15.824062 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:15.824033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:12:17.004207 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.004165    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"]
Sep 18 14:12:17.004606 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.004576    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f273ff05-db26-4335-8f95-04284b083607" containerName="maas-api"
Sep 18 14:12:17.004606 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.004589    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f273ff05-db26-4335-8f95-04284b083607" containerName="maas-api"
Sep 18 14:12:17.004695 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.004614    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="077ff23c-5df0-4623-a4c5-5899013850a2" containerName="payload-pre-processing"
Sep 18 14:12:17.004695 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.004620    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="077ff23c-5df0-4623-a4c5-5899013850a2" containerName="payload-pre-processing"
Sep 18 14:12:17.004695 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.004691    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f273ff05-db26-4335-8f95-04284b083607" containerName="maas-api"
Sep 18 14:12:17.004790 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.004701    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="077ff23c-5df0-4623-a4c5-5899013850a2" containerName="payload-pre-processing"
Sep 18 14:12:17.007232 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.007212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.009789 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.009761    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-93827d\"/\"default-dockercfg-nlc42\""
Sep 18 14:12:17.010642 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.010617    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-93827d\"/\"kube-root-ca.crt\""
Sep 18 14:12:17.010740 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.010655    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-93827d\"/\"simulator-w4-93827d-kserve-self-signed-certs\""
Sep 18 14:12:17.010740 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.010667    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-93827d\"/\"openshift-service-ca.crt\""
Sep 18 14:12:17.021744 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.021719    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"]
Sep 18 14:12:17.060070 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.060041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-model-cache\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.060215 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.060084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2dhf\" (UniqueName: \"kubernetes.io/projected/6cccf0cd-2042-4842-a2e6-471225fc7aac-kube-api-access-f2dhf\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.060215 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.060110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-tmp-dir\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.060343 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.060228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-dshm\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.060343 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.060259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-home\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.060343 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.060313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cccf0cd-2042-4842-a2e6-471225fc7aac-tls-certs\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.160626 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.160599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-dshm\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.160626 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.160626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-home\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.160784 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.160650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cccf0cd-2042-4842-a2e6-471225fc7aac-tls-certs\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.160784 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.160679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-model-cache\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.160784 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.160704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2dhf\" (UniqueName: \"kubernetes.io/projected/6cccf0cd-2042-4842-a2e6-471225fc7aac-kube-api-access-f2dhf\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.160784 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.160730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-tmp-dir\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.161093 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.161067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-home\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.161321 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.161095    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-model-cache\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.161439 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.161344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-tmp-dir\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.162991 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.162965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-dshm\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.163361 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.163344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cccf0cd-2042-4842-a2e6-471225fc7aac-tls-certs\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.181441 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.181411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2dhf\" (UniqueName: \"kubernetes.io/projected/6cccf0cd-2042-4842-a2e6-471225fc7aac-kube-api-access-f2dhf\") pod \"simulator-w4-93827d-kserve-567bd967bb-qrlnb\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") " pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.318054 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.317995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:17.590012 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.589983    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"]
Sep 18 14:12:17.591063 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:12:17.591039    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cccf0cd_2042_4842_a2e6_471225fc7aac.slice/crio-d533dcedffd4e57b74222af6653cf811342e50f559ac206df2b2af740cdd481e WatchSource:0}: Error finding container d533dcedffd4e57b74222af6653cf811342e50f559ac206df2b2af740cdd481e: Status 404 returned error can't find the container with id d533dcedffd4e57b74222af6653cf811342e50f559ac206df2b2af740cdd481e
Sep 18 14:12:17.769102 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.769066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb" event={"ID":"6cccf0cd-2042-4842-a2e6-471225fc7aac","Type":"ContainerStarted","Data":"87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6"}
Sep 18 14:12:17.769269 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.769112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb" event={"ID":"6cccf0cd-2042-4842-a2e6-471225fc7aac","Type":"ContainerStarted","Data":"d533dcedffd4e57b74222af6653cf811342e50f559ac206df2b2af740cdd481e"}
Sep 18 14:12:17.788170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:17.788108    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb" podStartSLOduration=1.788093701 podStartE2EDuration="1.788093701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:12:17.787003995 +0000 UTC m=+825.943524291" watchObservedRunningTime="2026-09-18 14:12:17.788093701 +0000 UTC m=+825.944613981"
Sep 18 14:12:27.318314 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:27.318275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:27.318314 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:27.318318    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:27.331604 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:27.331574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:27.847594 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:27.847562    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:12:31.546529 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:31.546492    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:12:40.662361 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:40.662325    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"]
Sep 18 14:12:40.665449 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:40.665430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"
Sep 18 14:12:40.667898 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:40.667875    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4fn9\""
Sep 18 14:12:40.668093 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:40.668076    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 14:12:40.676526 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:40.676501    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"]
Sep 18 14:12:40.760839 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:40.760808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb9c503-dfb9-41c4-ba78-28e73891cdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c\" (UID: \"bbb9c503-dfb9-41c4-ba78-28e73891cdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"
Sep 18 14:12:40.861827 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:40.861803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb9c503-dfb9-41c4-ba78-28e73891cdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c\" (UID: \"bbb9c503-dfb9-41c4-ba78-28e73891cdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"
Sep 18 14:12:40.862418 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:40.862401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb9c503-dfb9-41c4-ba78-28e73891cdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c\" (UID: \"bbb9c503-dfb9-41c4-ba78-28e73891cdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"
Sep 18 14:12:40.976207 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:40.976186    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"
Sep 18 14:12:41.246919 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:41.246889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"]
Sep 18 14:12:41.247976 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:12:41.247943    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb9c503_dfb9_41c4_ba78_28e73891cdd5.slice/crio-f75a9eed4efdc0e01a3cb4e4d34f2e61726e9a41d7437006ce98dad2d17d8c1f WatchSource:0}: Error finding container f75a9eed4efdc0e01a3cb4e4d34f2e61726e9a41d7437006ce98dad2d17d8c1f: Status 404 returned error can't find the container with id f75a9eed4efdc0e01a3cb4e4d34f2e61726e9a41d7437006ce98dad2d17d8c1f
Sep 18 14:12:41.932297 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:41.932265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c" event={"ID":"bbb9c503-dfb9-41c4-ba78-28e73891cdd5","Type":"ContainerStarted","Data":"f75a9eed4efdc0e01a3cb4e4d34f2e61726e9a41d7437006ce98dad2d17d8c1f"}
Sep 18 14:12:43.947116 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:43.947090    2563 generic.go:356] "Generic (PLEG): container finished" podID="bbb9c503-dfb9-41c4-ba78-28e73891cdd5" containerID="27989beefcef20a6f9d167a2eeb15164a6840b6e14075be0212141e1fc24c4fc" exitCode=0
Sep 18 14:12:43.947436 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:43.947167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c" event={"ID":"bbb9c503-dfb9-41c4-ba78-28e73891cdd5","Type":"ContainerDied","Data":"27989beefcef20a6f9d167a2eeb15164a6840b6e14075be0212141e1fc24c4fc"}
Sep 18 14:12:45.092079 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:45.092056    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"
Sep 18 14:12:45.197627 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:45.197600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb9c503-dfb9-41c4-ba78-28e73891cdd5-maas-api-service-ca\") pod \"bbb9c503-dfb9-41c4-ba78-28e73891cdd5\" (UID: \"bbb9c503-dfb9-41c4-ba78-28e73891cdd5\") "
Sep 18 14:12:45.197957 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:45.197933    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb9c503-dfb9-41c4-ba78-28e73891cdd5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bbb9c503-dfb9-41c4-ba78-28e73891cdd5" (UID: "bbb9c503-dfb9-41c4-ba78-28e73891cdd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:12:45.299217 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:45.299159    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb9c503-dfb9-41c4-ba78-28e73891cdd5-maas-api-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:12:45.960206 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:45.960177    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"
Sep 18 14:12:45.960206 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:45.960196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c" event={"ID":"bbb9c503-dfb9-41c4-ba78-28e73891cdd5","Type":"ContainerDied","Data":"f75a9eed4efdc0e01a3cb4e4d34f2e61726e9a41d7437006ce98dad2d17d8c1f"}
Sep 18 14:12:45.960434 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:45.960244    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f75a9eed4efdc0e01a3cb4e4d34f2e61726e9a41d7437006ce98dad2d17d8c1f"
Sep 18 14:12:48.527149 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.527095    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"]
Sep 18 14:12:48.527690 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.527668    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbb9c503-dfb9-41c4-ba78-28e73891cdd5" containerName="revoke-keys"
Sep 18 14:12:48.527779 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.527692    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbb9c503-dfb9-41c4-ba78-28e73891cdd5" containerName="revoke-keys"
Sep 18 14:12:48.527868 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.527854    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbb9c503-dfb9-41c4-ba78-28e73891cdd5" containerName="revoke-keys"
Sep 18 14:12:48.531011 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.530990    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.534782 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.534751    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dc2cee\"/\"kube-root-ca.crt\""
Sep 18 14:12:48.534902 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.534751    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dc2cee\"/\"openshift-service-ca.crt\""
Sep 18 14:12:48.534902 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.534839    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dc2cee\"/\"default-dockercfg-zstzr\""
Sep 18 14:12:48.534902 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.534858    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dc2cee\"/\"test-model-dc2cee-kserve-self-signed-certs\""
Sep 18 14:12:48.543381 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.543359    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"]
Sep 18 14:12:48.627443 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.627410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-dshm\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.627573 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.627446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-model-cache\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.627573 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.627528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c42fd9-e4e2-4c36-996f-78a964693ebe-tls-certs\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.627672 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.627569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-home\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.627672 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.627595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-tmp-dir\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.627672 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.627639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpq5z\" (UniqueName: \"kubernetes.io/projected/79c42fd9-e4e2-4c36-996f-78a964693ebe-kube-api-access-tpq5z\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.728750 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.728718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpq5z\" (UniqueName: \"kubernetes.io/projected/79c42fd9-e4e2-4c36-996f-78a964693ebe-kube-api-access-tpq5z\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.728851 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.728816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-dshm\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.728851 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.728842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-model-cache\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.728966 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.728887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c42fd9-e4e2-4c36-996f-78a964693ebe-tls-certs\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.729080 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.729065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-home\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.729160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.729098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-tmp-dir\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.729366 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.729344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-model-cache\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.729454 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.729437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-home\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.729511 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.729469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-tmp-dir\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.731152 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.731117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-dshm\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.731446 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.731429    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c42fd9-e4e2-4c36-996f-78a964693ebe-tls-certs\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.740297 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.740276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpq5z\" (UniqueName: \"kubernetes.io/projected/79c42fd9-e4e2-4c36-996f-78a964693ebe-kube-api-access-tpq5z\") pod \"test-model-dc2cee-kserve-668594485c-97s8p\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") " pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:48.840198 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:48.840110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:49.133969 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:49.133945    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"]
Sep 18 14:12:49.135504 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:12:49.135461    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79c42fd9_e4e2_4c36_996f_78a964693ebe.slice/crio-148e696bfabebef2626883b67eb69e145d4e008ac8f1b688117668c739293b08 WatchSource:0}: Error finding container 148e696bfabebef2626883b67eb69e145d4e008ac8f1b688117668c739293b08: Status 404 returned error can't find the container with id 148e696bfabebef2626883b67eb69e145d4e008ac8f1b688117668c739293b08
Sep 18 14:12:49.990338 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:49.990291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p" event={"ID":"79c42fd9-e4e2-4c36-996f-78a964693ebe","Type":"ContainerStarted","Data":"ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5"}
Sep 18 14:12:49.990785 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:49.990346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p" event={"ID":"79c42fd9-e4e2-4c36-996f-78a964693ebe","Type":"ContainerStarted","Data":"148e696bfabebef2626883b67eb69e145d4e008ac8f1b688117668c739293b08"}
Sep 18 14:12:50.010148 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:50.010074    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p" podStartSLOduration=2.010059444 podStartE2EDuration="2.010059444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:12:50.007398014 +0000 UTC m=+858.163918293" watchObservedRunningTime="2026-09-18 14:12:50.010059444 +0000 UTC m=+858.166579764"
Sep 18 14:12:51.216356 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:51.216318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:12:52.440240 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:52.440209    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"]
Sep 18 14:12:52.445756 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:52.445732    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5f7e430d-xmg4c"]
Sep 18 14:12:54.288325 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:54.288289    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbb9c503-dfb9-41c4-ba78-28e73891cdd5" path="/var/lib/kubelet/pods/bbb9c503-dfb9-41c4-ba78-28e73891cdd5/volumes"
Sep 18 14:12:58.840571 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:58.840524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:58.840571 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:58.840580    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:58.853385 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:58.853357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:12:59.057534 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:12:59.057507    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:13:14.014147 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:14.014081    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:13:19.559593 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.559561    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"]
Sep 18 14:13:19.560070 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.559769    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld" podUID="68a3f9ec-7666-43ec-9d35-3935637e885b" containerName="payload-processing" containerID="cri-o://a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7" gracePeriod=30
Sep 18 14:13:19.565164 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.565122    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"]
Sep 18 14:13:19.565503 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.565346    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj" podUID="2fa8de16-1e8c-4f2b-af20-5111aee62152" containerName="payload-pre-processing" containerID="cri-o://5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2" gracePeriod=30
Sep 18 14:13:19.834694 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.834569    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:13:19.851821 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.851798    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:13:19.903527 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.903502    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szzm8\" (UniqueName: \"kubernetes.io/projected/68a3f9ec-7666-43ec-9d35-3935637e885b-kube-api-access-szzm8\") pod \"68a3f9ec-7666-43ec-9d35-3935637e885b\" (UID: \"68a3f9ec-7666-43ec-9d35-3935637e885b\") "
Sep 18 14:13:19.903683 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.903604    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa8de16-1e8c-4f2b-af20-5111aee62152-plugins-config-volume\") pod \"2fa8de16-1e8c-4f2b-af20-5111aee62152\" (UID: \"2fa8de16-1e8c-4f2b-af20-5111aee62152\") "
Sep 18 14:13:19.903683 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.903638    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a3f9ec-7666-43ec-9d35-3935637e885b-plugins-config-volume\") pod \"68a3f9ec-7666-43ec-9d35-3935637e885b\" (UID: \"68a3f9ec-7666-43ec-9d35-3935637e885b\") "
Sep 18 14:13:19.903806 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.903749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ztn7\" (UniqueName: \"kubernetes.io/projected/2fa8de16-1e8c-4f2b-af20-5111aee62152-kube-api-access-6ztn7\") pod \"2fa8de16-1e8c-4f2b-af20-5111aee62152\" (UID: \"2fa8de16-1e8c-4f2b-af20-5111aee62152\") "
Sep 18 14:13:19.904043 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.904015    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa8de16-1e8c-4f2b-af20-5111aee62152-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2fa8de16-1e8c-4f2b-af20-5111aee62152" (UID: "2fa8de16-1e8c-4f2b-af20-5111aee62152"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:13:19.904188 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.904060    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a3f9ec-7666-43ec-9d35-3935637e885b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68a3f9ec-7666-43ec-9d35-3935637e885b" (UID: "68a3f9ec-7666-43ec-9d35-3935637e885b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:13:19.905667 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.905634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a3f9ec-7666-43ec-9d35-3935637e885b-kube-api-access-szzm8" (OuterVolumeSpecName: "kube-api-access-szzm8") pod "68a3f9ec-7666-43ec-9d35-3935637e885b" (UID: "68a3f9ec-7666-43ec-9d35-3935637e885b"). InnerVolumeSpecName "kube-api-access-szzm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:13:19.905743 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:19.905662    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa8de16-1e8c-4f2b-af20-5111aee62152-kube-api-access-6ztn7" (OuterVolumeSpecName: "kube-api-access-6ztn7") pod "2fa8de16-1e8c-4f2b-af20-5111aee62152" (UID: "2fa8de16-1e8c-4f2b-af20-5111aee62152"). InnerVolumeSpecName "kube-api-access-6ztn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:13:20.005491 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.005460    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ztn7\" (UniqueName: \"kubernetes.io/projected/2fa8de16-1e8c-4f2b-af20-5111aee62152-kube-api-access-6ztn7\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:13:20.005491 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.005487    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szzm8\" (UniqueName: \"kubernetes.io/projected/68a3f9ec-7666-43ec-9d35-3935637e885b-kube-api-access-szzm8\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:13:20.005648 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.005502    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fa8de16-1e8c-4f2b-af20-5111aee62152-plugins-config-volume\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:13:20.005648 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.005515    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68a3f9ec-7666-43ec-9d35-3935637e885b-plugins-config-volume\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:13:20.181674 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.181645    2563 generic.go:356] "Generic (PLEG): container finished" podID="2fa8de16-1e8c-4f2b-af20-5111aee62152" containerID="5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2" exitCode=0
Sep 18 14:13:20.181788 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.181687    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"
Sep 18 14:13:20.181788 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.181735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj" event={"ID":"2fa8de16-1e8c-4f2b-af20-5111aee62152","Type":"ContainerDied","Data":"5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2"}
Sep 18 14:13:20.181902 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.181788    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj" event={"ID":"2fa8de16-1e8c-4f2b-af20-5111aee62152","Type":"ContainerDied","Data":"4934b0a9e00eada24c88bca427b8d8e90038c414065ea1b47cacf2e7fd693d47"}
Sep 18 14:13:20.181902 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.181812    2563 scope.go:117] "RemoveContainer" containerID="5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2"
Sep 18 14:13:20.183167 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.183117    2563 generic.go:356] "Generic (PLEG): container finished" podID="68a3f9ec-7666-43ec-9d35-3935637e885b" containerID="a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7" exitCode=0
Sep 18 14:13:20.183264 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.183179    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"
Sep 18 14:13:20.183264 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.183190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld" event={"ID":"68a3f9ec-7666-43ec-9d35-3935637e885b","Type":"ContainerDied","Data":"a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7"}
Sep 18 14:13:20.183264 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.183227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld" event={"ID":"68a3f9ec-7666-43ec-9d35-3935637e885b","Type":"ContainerDied","Data":"7a873b208de3639c6bd27f4208f28d89651bf1340545142b1d539e138992ff8c"}
Sep 18 14:13:20.198847 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.198829    2563 scope.go:117] "RemoveContainer" containerID="5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2"
Sep 18 14:13:20.199109 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:13:20.199092    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2\": container with ID starting with 5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2 not found: ID does not exist" containerID="5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2"
Sep 18 14:13:20.199215 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.199114    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2"} err="failed to get container status \"5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2\": rpc error: code = NotFound desc = could not find container \"5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2\": container with ID starting with 5f632ac606b39060e39b3e0b8a54e4645ac495bbb1662c61d83216b2f41cd6a2 not found: ID does not exist"
Sep 18 14:13:20.199215 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.199161    2563 scope.go:117] "RemoveContainer" containerID="a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7"
Sep 18 14:13:20.210101 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.210083    2563 scope.go:117] "RemoveContainer" containerID="a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7"
Sep 18 14:13:20.210391 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:13:20.210373    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7\": container with ID starting with a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7 not found: ID does not exist" containerID="a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7"
Sep 18 14:13:20.210451 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.210396    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7"} err="failed to get container status \"a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7\": rpc error: code = NotFound desc = could not find container \"a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7\": container with ID starting with a550bd4785d25d43d6985b62a5015502c2e910eb0b83c61adf7ac2aea8de7df7 not found: ID does not exist"
Sep 18 14:13:20.215759 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.215736    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"]
Sep 18 14:13:20.219940 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.219916    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ae7021-cf568849d-44ppj"]
Sep 18 14:13:20.231195 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.231173    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"]
Sep 18 14:13:20.238086 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.238065    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ae7021-648bccd949-qvzld"]
Sep 18 14:13:20.289929 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.289900    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fa8de16-1e8c-4f2b-af20-5111aee62152" path="/var/lib/kubelet/pods/2fa8de16-1e8c-4f2b-af20-5111aee62152/volumes"
Sep 18 14:13:20.290295 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:20.290281    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68a3f9ec-7666-43ec-9d35-3935637e885b" path="/var/lib/kubelet/pods/68a3f9ec-7666-43ec-9d35-3935637e885b/volumes"
Sep 18 14:13:32.657530 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:32.657455    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:13:32.660749 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:32.660723    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:13:41.560678 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:13:41.560641    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:14:01.936622 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.936586    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"]
Sep 18 14:14:01.937096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.936988    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fa8de16-1e8c-4f2b-af20-5111aee62152" containerName="payload-pre-processing"
Sep 18 14:14:01.937096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.937000    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fa8de16-1e8c-4f2b-af20-5111aee62152" containerName="payload-pre-processing"
Sep 18 14:14:01.937096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.937013    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68a3f9ec-7666-43ec-9d35-3935637e885b" containerName="payload-processing"
Sep 18 14:14:01.937096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.937018    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="68a3f9ec-7666-43ec-9d35-3935637e885b" containerName="payload-processing"
Sep 18 14:14:01.937096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.937080    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fa8de16-1e8c-4f2b-af20-5111aee62152" containerName="payload-pre-processing"
Sep 18 14:14:01.937096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.937089    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="68a3f9ec-7666-43ec-9d35-3935637e885b" containerName="payload-processing"
Sep 18 14:14:01.940220 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.940200    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"
Sep 18 14:14:01.943219 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.943195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4fn9\""
Sep 18 14:14:01.944139 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.944107    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 14:14:01.951402 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.951378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62ec91b-0c01-4fcf-a689-80b9ca36c064-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7\" (UID: \"b62ec91b-0c01-4fcf-a689-80b9ca36c064\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"
Sep 18 14:14:01.956496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:01.956473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"]
Sep 18 14:14:02.052730 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:02.052704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62ec91b-0c01-4fcf-a689-80b9ca36c064-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7\" (UID: \"b62ec91b-0c01-4fcf-a689-80b9ca36c064\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"
Sep 18 14:14:02.053309 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:02.053290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62ec91b-0c01-4fcf-a689-80b9ca36c064-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7\" (UID: \"b62ec91b-0c01-4fcf-a689-80b9ca36c064\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"
Sep 18 14:14:02.250802 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:02.250739    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"
Sep 18 14:14:02.553322 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:14:02.553292    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62ec91b_0c01_4fcf_a689_80b9ca36c064.slice/crio-24f52f0b9e065b5da02e39d2953a114c35e34fd64ca016da46047e23a3e2815f WatchSource:0}: Error finding container 24f52f0b9e065b5da02e39d2953a114c35e34fd64ca016da46047e23a3e2815f: Status 404 returned error can't find the container with id 24f52f0b9e065b5da02e39d2953a114c35e34fd64ca016da46047e23a3e2815f
Sep 18 14:14:02.556806 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:02.556778    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"]
Sep 18 14:14:03.450112 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:03.450075    2563 generic.go:356] "Generic (PLEG): container finished" podID="b62ec91b-0c01-4fcf-a689-80b9ca36c064" containerID="862f06b558ba1ba2f99fb57f5e2b1fc71ca344848cde38a34f0dac3ce0edb7a6" exitCode=0
Sep 18 14:14:03.450423 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:03.450162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7" event={"ID":"b62ec91b-0c01-4fcf-a689-80b9ca36c064","Type":"ContainerDied","Data":"862f06b558ba1ba2f99fb57f5e2b1fc71ca344848cde38a34f0dac3ce0edb7a6"}
Sep 18 14:14:03.450423 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:03.450211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7" event={"ID":"b62ec91b-0c01-4fcf-a689-80b9ca36c064","Type":"ContainerStarted","Data":"24f52f0b9e065b5da02e39d2953a114c35e34fd64ca016da46047e23a3e2815f"}
Sep 18 14:14:03.971676 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:03.971644    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"]
Sep 18 14:14:03.971892 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:03.971869    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy" containerID="cri-o://5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd" gracePeriod=30
Sep 18 14:14:04.182619 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:04.182578    2563 patch_prober.go:28] interesting pod/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 14:14:04.182774 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:04.182637    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:14:04.583195 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:04.583172    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"
Sep 18 14:14:04.676685 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:04.676658    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62ec91b-0c01-4fcf-a689-80b9ca36c064-maas-api-service-ca\") pod \"b62ec91b-0c01-4fcf-a689-80b9ca36c064\" (UID: \"b62ec91b-0c01-4fcf-a689-80b9ca36c064\") "
Sep 18 14:14:04.677007 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:04.676986    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62ec91b-0c01-4fcf-a689-80b9ca36c064-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b62ec91b-0c01-4fcf-a689-80b9ca36c064" (UID: "b62ec91b-0c01-4fcf-a689-80b9ca36c064"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:14:04.778183 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:04.778104    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b62ec91b-0c01-4fcf-a689-80b9ca36c064-maas-api-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:14:05.463926 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:05.463888    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"
Sep 18 14:14:05.463926 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:05.463902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7" event={"ID":"b62ec91b-0c01-4fcf-a689-80b9ca36c064","Type":"ContainerDied","Data":"24f52f0b9e065b5da02e39d2953a114c35e34fd64ca016da46047e23a3e2815f"}
Sep 18 14:14:05.464169 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:05.463945    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24f52f0b9e065b5da02e39d2953a114c35e34fd64ca016da46047e23a3e2815f"
Sep 18 14:14:09.230666 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.230639    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:14:09.322686 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.322599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istiod-ca-cert\") pod \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") "
Sep 18 14:14:09.322686 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.322654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-credential-socket\") pod \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") "
Sep 18 14:14:09.322877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.322723    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cvbq\" (UniqueName: \"kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-kube-api-access-2cvbq\") pod \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") "
Sep 18 14:14:09.322877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.322752    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-envoy\") pod \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") "
Sep 18 14:14:09.322877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.322777    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-data\") pod \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") "
Sep 18 14:14:09.322877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.322816    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-socket\") pod \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") "
Sep 18 14:14:09.322877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.322846    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-certs\") pod \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") "
Sep 18 14:14:09.323110 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.322880    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-token\") pod \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") "
Sep 18 14:14:09.323110 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.322912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-podinfo\") pod \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\" (UID: \"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155\") "
Sep 18 14:14:09.323110 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.323013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" (UID: "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:14:09.323110 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.323106    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-data" (OuterVolumeSpecName: "istio-data") pod "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" (UID: "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:14:09.323326 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.323022    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" (UID: "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:14:09.323326 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.323225    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" (UID: "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:14:09.323436 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.323344    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-data\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:14:09.323436 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.323361    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:14:09.323436 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.323375    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istiod-ca-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:14:09.323436 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.323423    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-credential-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:14:09.323658 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.323449    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" (UID: "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:14:09.325196 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.325174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-token" (OuterVolumeSpecName: "istio-token") pod "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" (UID: "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:14:09.325306 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.325236    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" (UID: "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:14:09.325482 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.325458    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" (UID: "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:14:09.325529 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.325510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-kube-api-access-2cvbq" (OuterVolumeSpecName: "kube-api-access-2cvbq") pod "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" (UID: "bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155"). InnerVolumeSpecName "kube-api-access-2cvbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:14:09.423964 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.423921    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cvbq\" (UniqueName: \"kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-kube-api-access-2cvbq\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:14:09.423964 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.423957    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-envoy\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:14:09.423964 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.423968    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-workload-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:14:09.423964 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.423976    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-token\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:14:09.424282 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.423984    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155-istio-podinfo\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:14:09.489949 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.489912    2563 generic.go:356] "Generic (PLEG): container finished" podID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerID="5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd" exitCode=0
Sep 18 14:14:09.490148 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.489963    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"
Sep 18 14:14:09.490148 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.489988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" event={"ID":"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155","Type":"ContainerDied","Data":"5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd"}
Sep 18 14:14:09.490148 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.490047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb" event={"ID":"bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155","Type":"ContainerDied","Data":"9a52c9f1351de6b2c1b98f7853b7bb5b73e0a4a68c01bd330a0c303e68e6eae9"}
Sep 18 14:14:09.490148 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.490071    2563 scope.go:117] "RemoveContainer" containerID="5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd"
Sep 18 14:14:09.501157 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.501119    2563 scope.go:117] "RemoveContainer" containerID="d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa"
Sep 18 14:14:09.509601 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.509585    2563 scope.go:117] "RemoveContainer" containerID="5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd"
Sep 18 14:14:09.509869 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:14:09.509853    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd\": container with ID starting with 5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd not found: ID does not exist" containerID="5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd"
Sep 18 14:14:09.509914 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.509878    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd"} err="failed to get container status \"5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd\": rpc error: code = NotFound desc = could not find container \"5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd\": container with ID starting with 5731b0dc223b5244e08ff5ac6be958bbdc954a81f9bd7f61885d87212160f8dd not found: ID does not exist"
Sep 18 14:14:09.509914 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.509893    2563 scope.go:117] "RemoveContainer" containerID="d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa"
Sep 18 14:14:09.510201 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:14:09.510124    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa\": container with ID starting with d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa not found: ID does not exist" containerID="d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa"
Sep 18 14:14:09.510256 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.510209    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa"} err="failed to get container status \"d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa\": rpc error: code = NotFound desc = could not find container \"d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa\": container with ID starting with d36e2e824af1759ad20bae97b65630c150b3d34c21c37c4a2fa88c2baab8f2fa not found: ID does not exist"
Sep 18 14:14:09.528578 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.528552    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"]
Sep 18 14:14:09.537712 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:09.537689    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-ae7021-openshift-default-c8f6c5fc4-gdcxb"]
Sep 18 14:14:10.289258 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:10.289221    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" path="/var/lib/kubelet/pods/bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155/volumes"
Sep 18 14:14:13.219668 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.219636    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"]
Sep 18 14:14:13.220098 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.220035    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy"
Sep 18 14:14:13.220098 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.220049    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy"
Sep 18 14:14:13.220098 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.220074    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b62ec91b-0c01-4fcf-a689-80b9ca36c064" containerName="revoke-keys"
Sep 18 14:14:13.220098 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.220080    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b62ec91b-0c01-4fcf-a689-80b9ca36c064" containerName="revoke-keys"
Sep 18 14:14:13.220258 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.220163    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy"
Sep 18 14:14:13.220258 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.220171    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy"
Sep 18 14:14:13.220258 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.220178    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b62ec91b-0c01-4fcf-a689-80b9ca36c064" containerName="revoke-keys"
Sep 18 14:14:13.224969 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.224946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.227950 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.227920    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-e11b4c-serving-cert\""
Sep 18 14:14:13.228900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.228876    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5nh5f\""
Sep 18 14:14:13.229053 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.228878    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 14:14:13.229053 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.228974    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 14:14:13.229053 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.228883    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 14:14:13.237672 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.237598    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"]
Sep 18 14:14:13.360573 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.360529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-maas-api-tls\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.360763 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.360591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-metrics-certs\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.360763 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.360649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hcp8\" (UniqueName: \"kubernetes.io/projected/32d403cd-c156-44da-b589-d78640b00270-kube-api-access-6hcp8\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.360763 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.360702    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d403cd-c156-44da-b589-d78640b00270-openshift-service-ca\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.461276 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.461234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-maas-api-tls\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.461470 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.461294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-metrics-certs\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.461518 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.461454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcp8\" (UniqueName: \"kubernetes.io/projected/32d403cd-c156-44da-b589-d78640b00270-kube-api-access-6hcp8\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.461518 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.461499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d403cd-c156-44da-b589-d78640b00270-openshift-service-ca\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.462148 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.462112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d403cd-c156-44da-b589-d78640b00270-openshift-service-ca\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.463827 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.463803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-metrics-certs\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.463947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.463847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-maas-api-tls\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.470354 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.470291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hcp8\" (UniqueName: \"kubernetes.io/projected/32d403cd-c156-44da-b589-d78640b00270-kube-api-access-6hcp8\") pod \"maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.538757 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.538717    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:13.823445 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:13.823416    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"]
Sep 18 14:14:13.825755 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:14:13.825719    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d403cd_c156_44da_b589_d78640b00270.slice/crio-0c663f8327720a5a1ed8f291a2c469bda60662a339fe72617b74f82c08898e66 WatchSource:0}: Error finding container 0c663f8327720a5a1ed8f291a2c469bda60662a339fe72617b74f82c08898e66: Status 404 returned error can't find the container with id 0c663f8327720a5a1ed8f291a2c469bda60662a339fe72617b74f82c08898e66
Sep 18 14:14:14.476003 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:14.475962    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"]
Sep 18 14:14:14.481564 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:14.481530    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a721b5af-j5wj7"]
Sep 18 14:14:14.525581 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:14.525480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk" event={"ID":"32d403cd-c156-44da-b589-d78640b00270","Type":"ContainerStarted","Data":"f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a"}
Sep 18 14:14:14.525581 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:14.525550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk" event={"ID":"32d403cd-c156-44da-b589-d78640b00270","Type":"ContainerStarted","Data":"0c663f8327720a5a1ed8f291a2c469bda60662a339fe72617b74f82c08898e66"}
Sep 18 14:14:14.525581 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:14.525582    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:14.558010 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:14.557934    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk" podStartSLOduration=1.150584265 podStartE2EDuration="1.557909078s" totalImagesPullingTime="407.324813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:14:14.556328057 +0000 UTC m=+942.712848350" watchObservedRunningTime="2026-09-18 14:14:14.557909078 +0000 UTC m=+942.714429359"
Sep 18 14:14:16.288706 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:16.288667    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b62ec91b-0c01-4fcf-a689-80b9ca36c064" path="/var/lib/kubelet/pods/b62ec91b-0c01-4fcf-a689-80b9ca36c064/volumes"
Sep 18 14:14:20.537316 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:20.537284    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:14:22.314331 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:22.314286    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:14:33.281565 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.281519    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"]
Sep 18 14:14:33.282063 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.281949    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy"
Sep 18 14:14:33.282063 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.281963    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf82a9bd-e6e2-4cb7-ae7a-998baa8bf155" containerName="istio-proxy"
Sep 18 14:14:33.286706 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.286676    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.297169 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.297101    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4d9222\"/\"premium-w0-4d9222-kserve-self-signed-certs\""
Sep 18 14:14:33.309981 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.309946    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"]
Sep 18 14:14:33.470048 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.470004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tmp-dir\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.470283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.470062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-dshm\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.470283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.470169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tls-certs\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.470283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.470209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfcw\" (UniqueName: \"kubernetes.io/projected/0c78903b-eb41-4fe3-a8f3-98a48da8d959-kube-api-access-zxfcw\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.470283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.470248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-model-cache\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.470283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.470281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-home\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.571825 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.571720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tmp-dir\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.571825 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.571783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-dshm\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.572166 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.571852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tls-certs\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.572166 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.571890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfcw\" (UniqueName: \"kubernetes.io/projected/0c78903b-eb41-4fe3-a8f3-98a48da8d959-kube-api-access-zxfcw\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.572166 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.571931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-model-cache\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.572166 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.571963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-home\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.572416 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.572304    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tmp-dir\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.572416 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.572334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-model-cache\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.572416 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.572386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-home\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.574222 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.574199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-dshm\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.574500 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.574480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tls-certs\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.581249 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.581224    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfcw\" (UniqueName: \"kubernetes.io/projected/0c78903b-eb41-4fe3-a8f3-98a48da8d959-kube-api-access-zxfcw\") pod \"premium-w0-4d9222-kserve-5dd89f6577-wdvzx\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") " pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.596976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.596945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:33.877032 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:33.876999    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"]
Sep 18 14:14:33.878855 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:14:33.878821    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c78903b_eb41_4fe3_a8f3_98a48da8d959.slice/crio-53f3595487092b2483d6a1cfd29d970cd94e070d70b4c7fc6ea3cc5b79172d39 WatchSource:0}: Error finding container 53f3595487092b2483d6a1cfd29d970cd94e070d70b4c7fc6ea3cc5b79172d39: Status 404 returned error can't find the container with id 53f3595487092b2483d6a1cfd29d970cd94e070d70b4c7fc6ea3cc5b79172d39
Sep 18 14:14:34.646405 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:34.646361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx" event={"ID":"0c78903b-eb41-4fe3-a8f3-98a48da8d959","Type":"ContainerStarted","Data":"eb69cfae152d0ec359e5aa0eabfcacadaf39641ceb71474d81f5966e34ae73cd"}
Sep 18 14:14:34.646405 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:34.646409    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx" event={"ID":"0c78903b-eb41-4fe3-a8f3-98a48da8d959","Type":"ContainerStarted","Data":"53f3595487092b2483d6a1cfd29d970cd94e070d70b4c7fc6ea3cc5b79172d39"}
Sep 18 14:14:34.672964 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:34.672906    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx" podStartSLOduration=1.672888279 podStartE2EDuration="1.672888279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:14:34.6689738 +0000 UTC m=+962.825494082" watchObservedRunningTime="2026-09-18 14:14:34.672888279 +0000 UTC m=+962.829408560"
Sep 18 14:14:41.275467 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:41.275432    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:14:43.597722 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:43.597675    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:43.598181 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:43.597737    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:43.610640 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:43.610610    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:14:43.711275 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:14:43.711241    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:15:00.170997 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.170900    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm"]
Sep 18 14:15:00.174798 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.174771    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829015-jthml"]
Sep 18 14:15:00.174956 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.174908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm"
Sep 18 14:15:00.178380 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.178353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829015-jthml"
Sep 18 14:15:00.182768 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.182743    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-l4fn9\""
Sep 18 14:15:00.189292 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.189266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm"
Sep 18 14:15:00.195987 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.195853    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm"]
Sep 18 14:15:00.199211 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.199172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829015-jthml"]
Sep 18 14:15:00.200316 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.200287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829015-jthml"
Sep 18 14:15:00.209728 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.209689    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss"]
Sep 18 14:15:00.214266 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.214243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss"
Sep 18 14:15:00.221747 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.221714    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss"]
Sep 18 14:15:00.250031 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.249998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss"
Sep 18 14:15:00.502487 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.502449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm"]
Sep 18 14:15:00.504095 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:15:00.504072    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616b2faf_4159_4c2d_bd22_62296ae23ccf.slice/crio-fad49447f581c44763bbeb49b392ec6d13e0c911e09c9c81fc5455103dbc8b84 WatchSource:0}: Error finding container fad49447f581c44763bbeb49b392ec6d13e0c911e09c9c81fc5455103dbc8b84: Status 404 returned error can't find the container with id fad49447f581c44763bbeb49b392ec6d13e0c911e09c9c81fc5455103dbc8b84
Sep 18 14:15:00.509046 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.509023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829015-jthml"]
Sep 18 14:15:00.510775 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:15:00.510749    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725a8a0f_50f4_443d_b874_fe929e24093e.slice/crio-496ff2ebc9581d7d30d0da68e405edb8d1f89254edb7200c4b6bd3aced20ea75 WatchSource:0}: Error finding container 496ff2ebc9581d7d30d0da68e405edb8d1f89254edb7200c4b6bd3aced20ea75: Status 404 returned error can't find the container with id 496ff2ebc9581d7d30d0da68e405edb8d1f89254edb7200c4b6bd3aced20ea75
Sep 18 14:15:00.558704 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.557767    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss"]
Sep 18 14:15:00.805078 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.804992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829015-jthml" event={"ID":"725a8a0f-50f4-443d-b874-fe929e24093e","Type":"ContainerStarted","Data":"496ff2ebc9581d7d30d0da68e405edb8d1f89254edb7200c4b6bd3aced20ea75"}
Sep 18 14:15:00.806600 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.806570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm" event={"ID":"616b2faf-4159-4c2d-bd22-62296ae23ccf","Type":"ContainerStarted","Data":"fad49447f581c44763bbeb49b392ec6d13e0c911e09c9c81fc5455103dbc8b84"}
Sep 18 14:15:00.808092 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:00.808065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss" event={"ID":"650bc3c6-851e-4c11-95b6-30ea3a0f3c64","Type":"ContainerStarted","Data":"50da08dc8642632eaf7ef3209225259d1572b26818d4d71370a4fa9ee32b563a"}
Sep 18 14:15:01.816381 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:01.816295    2563 generic.go:356] "Generic (PLEG): container finished" podID="725a8a0f-50f4-443d-b874-fe929e24093e" containerID="e50afe59e9a8cf532b1769b9b0d47393467254e21306ee4b783d1312c27781d0" exitCode=0
Sep 18 14:15:01.816787 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:01.816378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829015-jthml" event={"ID":"725a8a0f-50f4-443d-b874-fe929e24093e","Type":"ContainerDied","Data":"e50afe59e9a8cf532b1769b9b0d47393467254e21306ee4b783d1312c27781d0"}
Sep 18 14:15:01.818222 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:01.818191    2563 generic.go:356] "Generic (PLEG): container finished" podID="616b2faf-4159-4c2d-bd22-62296ae23ccf" containerID="19e8f5d6d5060826cb4ab7ede0e6c37c5de4ab574a4d045c701d4585890481a4" exitCode=0
Sep 18 14:15:01.818352 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:01.818274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm" event={"ID":"616b2faf-4159-4c2d-bd22-62296ae23ccf","Type":"ContainerDied","Data":"19e8f5d6d5060826cb4ab7ede0e6c37c5de4ab574a4d045c701d4585890481a4"}
Sep 18 14:15:01.819964 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:01.819943    2563 generic.go:356] "Generic (PLEG): container finished" podID="650bc3c6-851e-4c11-95b6-30ea3a0f3c64" containerID="187dbece35a48c4a9855118c39fe8d3681ec53058d0b4b7394731927633eda44" exitCode=0
Sep 18 14:15:01.820070 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:01.820006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss" event={"ID":"650bc3c6-851e-4c11-95b6-30ea3a0f3c64","Type":"ContainerDied","Data":"187dbece35a48c4a9855118c39fe8d3681ec53058d0b4b7394731927633eda44"}
Sep 18 14:15:01.922291 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:01.922248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:15:02.980865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:02.980838    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss"
Sep 18 14:15:03.026674 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:03.026645    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm"
Sep 18 14:15:03.029832 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:03.029809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829015-jthml"
Sep 18 14:15:03.833913 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:03.833881    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss"
Sep 18 14:15:03.833913 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:03.833903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss" event={"ID":"650bc3c6-851e-4c11-95b6-30ea3a0f3c64","Type":"ContainerDied","Data":"50da08dc8642632eaf7ef3209225259d1572b26818d4d71370a4fa9ee32b563a"}
Sep 18 14:15:03.834207 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:03.833956    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50da08dc8642632eaf7ef3209225259d1572b26818d4d71370a4fa9ee32b563a"
Sep 18 14:15:03.835296 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:03.835278    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829015-jthml"
Sep 18 14:15:03.835406 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:03.835324    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829015-jthml" event={"ID":"725a8a0f-50f4-443d-b874-fe929e24093e","Type":"ContainerDied","Data":"496ff2ebc9581d7d30d0da68e405edb8d1f89254edb7200c4b6bd3aced20ea75"}
Sep 18 14:15:03.835406 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:03.835377    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="496ff2ebc9581d7d30d0da68e405edb8d1f89254edb7200c4b6bd3aced20ea75"
Sep 18 14:15:03.836754 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:03.836732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm" event={"ID":"616b2faf-4159-4c2d-bd22-62296ae23ccf","Type":"ContainerDied","Data":"fad49447f581c44763bbeb49b392ec6d13e0c911e09c9c81fc5455103dbc8b84"}
Sep 18 14:15:03.836880 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:03.836775    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad49447f581c44763bbeb49b392ec6d13e0c911e09c9c81fc5455103dbc8b84"
Sep 18 14:15:03.836880 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:03.836738    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm"
Sep 18 14:15:19.232021 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:19.231978    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:15:26.514494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.514444    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"]
Sep 18 14:15:26.514899 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.514864    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616b2faf-4159-4c2d-bd22-62296ae23ccf" containerName="cleanup"
Sep 18 14:15:26.514899 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.514876    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="616b2faf-4159-4c2d-bd22-62296ae23ccf" containerName="cleanup"
Sep 18 14:15:26.514899 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.514897    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725a8a0f-50f4-443d-b874-fe929e24093e" containerName="cleanup"
Sep 18 14:15:26.515007 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.514903    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="725a8a0f-50f4-443d-b874-fe929e24093e" containerName="cleanup"
Sep 18 14:15:26.515007 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.514912    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="650bc3c6-851e-4c11-95b6-30ea3a0f3c64" containerName="cleanup"
Sep 18 14:15:26.515007 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.514918    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="650bc3c6-851e-4c11-95b6-30ea3a0f3c64" containerName="cleanup"
Sep 18 14:15:26.515007 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.514979    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="650bc3c6-851e-4c11-95b6-30ea3a0f3c64" containerName="cleanup"
Sep 18 14:15:26.515007 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.514987    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="616b2faf-4159-4c2d-bd22-62296ae23ccf" containerName="cleanup"
Sep 18 14:15:26.515007 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.514995    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="725a8a0f-50f4-443d-b874-fe929e24093e" containerName="cleanup"
Sep 18 14:15:26.518508 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.518486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.522307 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.522278    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e11b4c\"/\"kube-root-ca.crt\""
Sep 18 14:15:26.522475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.522280    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-e11b4c\"/\"openshift-service-ca.crt\""
Sep 18 14:15:26.522475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.522288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e11b4c\"/\"default-dockercfg-tltn8\""
Sep 18 14:15:26.522475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.522417    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-e11b4c\"/\"explicit-model-e11b4c-kserve-self-signed-certs\""
Sep 18 14:15:26.528073 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.528034    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"]
Sep 18 14:15:26.595080 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.595037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-model-cache\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.595298 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.595098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b4ca49-079a-42f9-939e-81cc43c582ed-tls-certs\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.595298 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.595181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tgxg\" (UniqueName: \"kubernetes.io/projected/39b4ca49-079a-42f9-939e-81cc43c582ed-kube-api-access-4tgxg\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.595298 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.595269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-tmp-dir\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.595440 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.595313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-dshm\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.595440 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.595379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-home\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.696625 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.696585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-tmp-dir\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.696828 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.696655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-dshm\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.696828 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.696684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-home\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.696828 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.696714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-model-cache\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.697007 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.696898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b4ca49-079a-42f9-939e-81cc43c582ed-tls-certs\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.697007 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.696947    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tgxg\" (UniqueName: \"kubernetes.io/projected/39b4ca49-079a-42f9-939e-81cc43c582ed-kube-api-access-4tgxg\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.697184 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.697124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-home\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.697287 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.697265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-model-cache\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.697411 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.697389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-tmp-dir\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.699050 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.699030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-dshm\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.699505 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.699484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b4ca49-079a-42f9-939e-81cc43c582ed-tls-certs\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.705357 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.705322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tgxg\" (UniqueName: \"kubernetes.io/projected/39b4ca49-079a-42f9-939e-81cc43c582ed-kube-api-access-4tgxg\") pod \"explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") " pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:26.829946 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:26.829832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:27.113039 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:27.113006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"]
Sep 18 14:15:27.114230 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:15:27.114199    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39b4ca49_079a_42f9_939e_81cc43c582ed.slice/crio-76e776989b695f8ff6d305c3a06b6d71a31b3e46b170fb958c9123af088fa1ac WatchSource:0}: Error finding container 76e776989b695f8ff6d305c3a06b6d71a31b3e46b170fb958c9123af088fa1ac: Status 404 returned error can't find the container with id 76e776989b695f8ff6d305c3a06b6d71a31b3e46b170fb958c9123af088fa1ac
Sep 18 14:15:27.994900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:27.994855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq" event={"ID":"39b4ca49-079a-42f9-939e-81cc43c582ed","Type":"ContainerStarted","Data":"1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3"}
Sep 18 14:15:27.995344 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:27.994916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq" event={"ID":"39b4ca49-079a-42f9-939e-81cc43c582ed","Type":"ContainerStarted","Data":"76e776989b695f8ff6d305c3a06b6d71a31b3e46b170fb958c9123af088fa1ac"}
Sep 18 14:15:28.013636 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:28.013573    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq" podStartSLOduration=2.01355079 podStartE2EDuration="2.01355079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:15:28.013102738 +0000 UTC m=+1016.169623020" watchObservedRunningTime="2026-09-18 14:15:28.01355079 +0000 UTC m=+1016.170071072"
Sep 18 14:15:36.830497 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:36.830431    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:36.830497 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:36.830503    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:36.843643 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:36.843595    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:37.031670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:37.031628    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:15:37.068028 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:37.067991    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:15:50.843833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:50.843790    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"]
Sep 18 14:15:59.918976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:59.918934    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"]
Sep 18 14:15:59.922675 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:59.922654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:15:59.926620 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:59.926596    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-93827d\"/\"e2e-conflict-a-1b2f9999-kserve-self-signed-certs\""
Sep 18 14:15:59.936998 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:59.936971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"]
Sep 18 14:15:59.943183 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:59.943148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-dshm\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:15:59.943327 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:59.943221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tls-certs\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:15:59.943327 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:59.943282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-home\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:15:59.943327 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:59.943311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tmp-dir\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:15:59.943436 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:59.943352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-model-cache\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:15:59.943436 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:15:59.943418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mf6z\" (UniqueName: \"kubernetes.io/projected/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-kube-api-access-2mf6z\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.044573 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.044520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-dshm\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.044573 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.044581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tls-certs\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.044872 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.044621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-home\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.044872 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.044644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tmp-dir\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.044872 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.044681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-model-cache\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.044872 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.044729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mf6z\" (UniqueName: \"kubernetes.io/projected/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-kube-api-access-2mf6z\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.045146 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.045105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-home\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.045258 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.045233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tmp-dir\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.045320 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.045262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-model-cache\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.047156 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.047120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-dshm\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.047582 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.047563    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tls-certs\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.055543 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.055520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mf6z\" (UniqueName: \"kubernetes.io/projected/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-kube-api-access-2mf6z\") pod \"e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.232596 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.232550    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:00.514886 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:00.514848    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"]
Sep 18 14:16:00.516815 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:16:00.516786    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode06b7348_4fa2_4ea5_ae0e_6b5dd621943a.slice/crio-1a3e71270239ec917b5703f869853d6c7e4c76d71555198c5bc4001b7316f053 WatchSource:0}: Error finding container 1a3e71270239ec917b5703f869853d6c7e4c76d71555198c5bc4001b7316f053: Status 404 returned error can't find the container with id 1a3e71270239ec917b5703f869853d6c7e4c76d71555198c5bc4001b7316f053
Sep 18 14:16:01.211833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:01.211790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq" event={"ID":"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a","Type":"ContainerStarted","Data":"be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0"}
Sep 18 14:16:01.211833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:01.211842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq" event={"ID":"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a","Type":"ContainerStarted","Data":"1a3e71270239ec917b5703f869853d6c7e4c76d71555198c5bc4001b7316f053"}
Sep 18 14:16:01.252361 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:01.252299    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq" podStartSLOduration=2.2522785770000002 podStartE2EDuration="2.252278577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:16:01.250743337 +0000 UTC m=+1049.407263618" watchObservedRunningTime="2026-09-18 14:16:01.252278577 +0000 UTC m=+1049.408798935"
Sep 18 14:16:05.864192 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:05.864124    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq" podUID="39b4ca49-079a-42f9-939e-81cc43c582ed" containerName="main" containerID="cri-o://1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3" gracePeriod=45
Sep 18 14:16:06.136770 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.136733    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:16:06.211243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.211206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-home\") pod \"39b4ca49-079a-42f9-939e-81cc43c582ed\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") "
Sep 18 14:16:06.211463 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.211268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-tmp-dir\") pod \"39b4ca49-079a-42f9-939e-81cc43c582ed\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") "
Sep 18 14:16:06.211463 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.211325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-model-cache\") pod \"39b4ca49-079a-42f9-939e-81cc43c582ed\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") "
Sep 18 14:16:06.211463 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.211377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tgxg\" (UniqueName: \"kubernetes.io/projected/39b4ca49-079a-42f9-939e-81cc43c582ed-kube-api-access-4tgxg\") pod \"39b4ca49-079a-42f9-939e-81cc43c582ed\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") "
Sep 18 14:16:06.211654 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.211494    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b4ca49-079a-42f9-939e-81cc43c582ed-tls-certs\") pod \"39b4ca49-079a-42f9-939e-81cc43c582ed\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") "
Sep 18 14:16:06.211654 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.211542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-dshm\") pod \"39b4ca49-079a-42f9-939e-81cc43c582ed\" (UID: \"39b4ca49-079a-42f9-939e-81cc43c582ed\") "
Sep 18 14:16:06.211654 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.211579    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-home" (OuterVolumeSpecName: "home") pod "39b4ca49-079a-42f9-939e-81cc43c582ed" (UID: "39b4ca49-079a-42f9-939e-81cc43c582ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:06.211796 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.211682    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39b4ca49-079a-42f9-939e-81cc43c582ed" (UID: "39b4ca49-079a-42f9-939e-81cc43c582ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:06.211940 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.211918    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:16:06.212029 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.211942    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:16:06.212228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.212139    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "39b4ca49-079a-42f9-939e-81cc43c582ed" (UID: "39b4ca49-079a-42f9-939e-81cc43c582ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:06.214738 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.214712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b4ca49-079a-42f9-939e-81cc43c582ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39b4ca49-079a-42f9-939e-81cc43c582ed" (UID: "39b4ca49-079a-42f9-939e-81cc43c582ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:16:06.214738 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.214712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-dshm" (OuterVolumeSpecName: "dshm") pod "39b4ca49-079a-42f9-939e-81cc43c582ed" (UID: "39b4ca49-079a-42f9-939e-81cc43c582ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:16:06.214949 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.214927    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b4ca49-079a-42f9-939e-81cc43c582ed-kube-api-access-4tgxg" (OuterVolumeSpecName: "kube-api-access-4tgxg") pod "39b4ca49-079a-42f9-939e-81cc43c582ed" (UID: "39b4ca49-079a-42f9-939e-81cc43c582ed"). InnerVolumeSpecName "kube-api-access-4tgxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:06.249047 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.249009    2563 generic.go:356] "Generic (PLEG): container finished" podID="39b4ca49-079a-42f9-939e-81cc43c582ed" containerID="1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3" exitCode=0
Sep 18 14:16:06.249283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.249066    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"
Sep 18 14:16:06.249283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.249096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq" event={"ID":"39b4ca49-079a-42f9-939e-81cc43c582ed","Type":"ContainerDied","Data":"1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3"}
Sep 18 14:16:06.249283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.249182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq" event={"ID":"39b4ca49-079a-42f9-939e-81cc43c582ed","Type":"ContainerDied","Data":"76e776989b695f8ff6d305c3a06b6d71a31b3e46b170fb958c9123af088fa1ac"}
Sep 18 14:16:06.249283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.249207    2563 scope.go:117] "RemoveContainer" containerID="1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3"
Sep 18 14:16:06.262796 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.262765    2563 scope.go:117] "RemoveContainer" containerID="1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3"
Sep 18 14:16:06.263119 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:16:06.263101    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3\": container with ID starting with 1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3 not found: ID does not exist" containerID="1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3"
Sep 18 14:16:06.263194 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.263158    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3"} err="failed to get container status \"1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3\": rpc error: code = NotFound desc = could not find container \"1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3\": container with ID starting with 1b3067acdff73220481c0e2c1c8c2b1950f69540f4ed3d3f01c0fe05bef4e7f3 not found: ID does not exist"
Sep 18 14:16:06.279957 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.279917    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"]
Sep 18 14:16:06.299468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.299432    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-e11b4c/explicit-model-e11b4c-kserve-7d4c69bcfc-2gltq"]
Sep 18 14:16:06.313375 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.313335    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b4ca49-079a-42f9-939e-81cc43c582ed-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:16:06.313547 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.313438    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:16:06.313547 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.313449    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39b4ca49-079a-42f9-939e-81cc43c582ed-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:16:06.313547 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:06.313459    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tgxg\" (UniqueName: \"kubernetes.io/projected/39b4ca49-079a-42f9-939e-81cc43c582ed-kube-api-access-4tgxg\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:16:07.140117 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.140077    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"]
Sep 18 14:16:07.140707 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.140402    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk" podUID="32d403cd-c156-44da-b589-d78640b00270" containerName="maas-api" containerID="cri-o://f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a" gracePeriod=30
Sep 18 14:16:07.182313 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.182272    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm"]
Sep 18 14:16:07.186571 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.186540    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-e11b4c-29829015-xtlrm"]
Sep 18 14:16:07.414931 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.414897    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:16:07.526367 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.526320    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-metrics-certs\") pod \"32d403cd-c156-44da-b589-d78640b00270\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") "
Sep 18 14:16:07.526367 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.526373    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hcp8\" (UniqueName: \"kubernetes.io/projected/32d403cd-c156-44da-b589-d78640b00270-kube-api-access-6hcp8\") pod \"32d403cd-c156-44da-b589-d78640b00270\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") "
Sep 18 14:16:07.526647 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.526423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d403cd-c156-44da-b589-d78640b00270-openshift-service-ca\") pod \"32d403cd-c156-44da-b589-d78640b00270\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") "
Sep 18 14:16:07.526647 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.526475    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-maas-api-tls\") pod \"32d403cd-c156-44da-b589-d78640b00270\" (UID: \"32d403cd-c156-44da-b589-d78640b00270\") "
Sep 18 14:16:07.526936 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.526905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d403cd-c156-44da-b589-d78640b00270-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32d403cd-c156-44da-b589-d78640b00270" (UID: "32d403cd-c156-44da-b589-d78640b00270"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:16:07.528975 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.528938    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "32d403cd-c156-44da-b589-d78640b00270" (UID: "32d403cd-c156-44da-b589-d78640b00270"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:16:07.529075 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.528970    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d403cd-c156-44da-b589-d78640b00270-kube-api-access-6hcp8" (OuterVolumeSpecName: "kube-api-access-6hcp8") pod "32d403cd-c156-44da-b589-d78640b00270" (UID: "32d403cd-c156-44da-b589-d78640b00270"). InnerVolumeSpecName "kube-api-access-6hcp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:16:07.529075 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.528992    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32d403cd-c156-44da-b589-d78640b00270" (UID: "32d403cd-c156-44da-b589-d78640b00270"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:16:07.628282 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.628239    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-maas-api-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:16:07.628282 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.628273    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d403cd-c156-44da-b589-d78640b00270-metrics-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:16:07.628282 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.628282    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hcp8\" (UniqueName: \"kubernetes.io/projected/32d403cd-c156-44da-b589-d78640b00270-kube-api-access-6hcp8\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:16:07.628282 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:07.628292    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d403cd-c156-44da-b589-d78640b00270-openshift-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:16:08.037290 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.037246    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:16:08.267553 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.267512    2563 generic.go:356] "Generic (PLEG): container finished" podID="32d403cd-c156-44da-b589-d78640b00270" containerID="f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a" exitCode=0
Sep 18 14:16:08.267988 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.267560    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"
Sep 18 14:16:08.267988 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.267600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk" event={"ID":"32d403cd-c156-44da-b589-d78640b00270","Type":"ContainerDied","Data":"f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a"}
Sep 18 14:16:08.267988 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.267696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk" event={"ID":"32d403cd-c156-44da-b589-d78640b00270","Type":"ContainerDied","Data":"0c663f8327720a5a1ed8f291a2c469bda60662a339fe72617b74f82c08898e66"}
Sep 18 14:16:08.267988 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.267727    2563 scope.go:117] "RemoveContainer" containerID="f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a"
Sep 18 14:16:08.278567 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.278542    2563 scope.go:117] "RemoveContainer" containerID="f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a"
Sep 18 14:16:08.278905 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:16:08.278883    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a\": container with ID starting with f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a not found: ID does not exist" containerID="f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a"
Sep 18 14:16:08.279002 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.278920    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a"} err="failed to get container status \"f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a\": rpc error: code = NotFound desc = could not find container \"f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a\": container with ID starting with f7909ffd50eff44a4676805e80d328bb7e25f766b7bdea1d6755dde9498ab46a not found: ID does not exist"
Sep 18 14:16:08.289267 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.289195    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b4ca49-079a-42f9-939e-81cc43c582ed" path="/var/lib/kubelet/pods/39b4ca49-079a-42f9-939e-81cc43c582ed/volumes"
Sep 18 14:16:08.289546 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.289534    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="616b2faf-4159-4c2d-bd22-62296ae23ccf" path="/var/lib/kubelet/pods/616b2faf-4159-4c2d-bd22-62296ae23ccf/volumes"
Sep 18 14:16:08.317200 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.317164    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"]
Sep 18 14:16:08.328040 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:08.328004    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-e11b4c-7ff64f98c4-l5tlk"]
Sep 18 14:16:10.233077 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:10.233040    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:10.233651 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:10.233099    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:10.246113 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:10.246082    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:10.289064 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:10.289016    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d403cd-c156-44da-b589-d78640b00270" path="/var/lib/kubelet/pods/32d403cd-c156-44da-b589-d78640b00270/volumes"
Sep 18 14:16:10.293561 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:10.293530    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:16:16.108656 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.108607    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"]
Sep 18 14:16:16.109363 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.109338    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b4ca49-079a-42f9-939e-81cc43c582ed" containerName="main"
Sep 18 14:16:16.109474 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.109365    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b4ca49-079a-42f9-939e-81cc43c582ed" containerName="main"
Sep 18 14:16:16.109474 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.109411    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d403cd-c156-44da-b589-d78640b00270" containerName="maas-api"
Sep 18 14:16:16.109474 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.109420    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d403cd-c156-44da-b589-d78640b00270" containerName="maas-api"
Sep 18 14:16:16.109629 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.109535    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d403cd-c156-44da-b589-d78640b00270" containerName="maas-api"
Sep 18 14:16:16.109629 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.109554    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b4ca49-079a-42f9-939e-81cc43c582ed" containerName="main"
Sep 18 14:16:16.115010 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.114979    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.123360 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.123325    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-e8ad4e-openshift-default-dockercfg-twv6t\""
Sep 18 14:16:16.147660 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.147604    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"]
Sep 18 14:16:16.223020 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.222974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18484eb-902e-4e1c-8eb4-a622e13a3110-istiod-ca-cert\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.223020 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.223023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-podinfo\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.223346 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.223048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-envoy\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.223346 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.223185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-certs\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.223346 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.223238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-token\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.223513 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.223346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-socket\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.223513 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.223395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-credential-socket\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.223513 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.223469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdvfg\" (UniqueName: \"kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-kube-api-access-pdvfg\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.223513 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.223504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-data\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.324496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.324449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-certs\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.324721 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.324517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-token\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.324721 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.324575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-socket\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.324721 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.324628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-credential-socket\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.324721 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.324688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdvfg\" (UniqueName: \"kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-kube-api-access-pdvfg\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.324721 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.324719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-data\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.325016 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.324780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18484eb-902e-4e1c-8eb4-a622e13a3110-istiod-ca-cert\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.325016 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.324814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-podinfo\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.325016 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.324844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-envoy\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.325016 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.324972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-certs\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.325285 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.325228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-socket\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.325345 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.325282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-data\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.325345 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.325332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-credential-socket\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.325782 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.325754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18484eb-902e-4e1c-8eb4-a622e13a3110-istiod-ca-cert\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.327711 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.327679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-envoy\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.328087 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.328068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-podinfo\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.340758 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.340727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-token\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.345805 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.345769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdvfg\" (UniqueName: \"kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-kube-api-access-pdvfg\") pod \"e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") " pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.426480 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.426428    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:16.721409 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.721378    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"]
Sep 18 14:16:16.722931 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:16:16.722898    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18484eb_902e_4e1c_8eb4_a622e13a3110.slice/crio-486e3ba2fba17943bf34f2ea4948e18087381e11543981e221382d1558a85af5 WatchSource:0}: Error finding container 486e3ba2fba17943bf34f2ea4948e18087381e11543981e221382d1558a85af5: Status 404 returned error can't find the container with id 486e3ba2fba17943bf34f2ea4948e18087381e11543981e221382d1558a85af5
Sep 18 14:16:16.725579 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.725537    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:16:16.725640 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.725625    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:16:16.725673 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:16.725656    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:16:17.334824 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:17.334772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk" event={"ID":"c18484eb-902e-4e1c-8eb4-a622e13a3110","Type":"ContainerStarted","Data":"1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70"}
Sep 18 14:16:17.335302 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:17.334860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk" event={"ID":"c18484eb-902e-4e1c-8eb4-a622e13a3110","Type":"ContainerStarted","Data":"486e3ba2fba17943bf34f2ea4948e18087381e11543981e221382d1558a85af5"}
Sep 18 14:16:17.360685 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:17.360619    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk" podStartSLOduration=1.360600204 podStartE2EDuration="1.360600204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:16:17.357055718 +0000 UTC m=+1065.513576026" watchObservedRunningTime="2026-09-18 14:16:17.360600204 +0000 UTC m=+1065.517120486"
Sep 18 14:16:17.427194 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:17.427151    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:17.429364 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:17.429325    2563 patch_prober.go:28] interesting pod/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 18 14:16:17.429553 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:17.429406    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk" podUID="c18484eb-902e-4e1c-8eb4-a622e13a3110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 18 14:16:18.432255 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:18.432216    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:19.325144 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:19.325101    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:16:19.347902 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:19.347869    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:19.349092 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:19.349067    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:16:29.027972 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.027933    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"]
Sep 18 14:16:29.031902 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.031881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.037406 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.037347    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e8ad4e\"/\"default-dockercfg-5qtg4\""
Sep 18 14:16:29.037406 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.037374    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e8ad4e\"/\"kube-root-ca.crt\""
Sep 18 14:16:29.037667 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.037531    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e8ad4e\"/\"noten-model-e8ad4e-kserve-self-signed-certs\""
Sep 18 14:16:29.037667 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.037636    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e8ad4e\"/\"openshift-service-ca.crt\""
Sep 18 14:16:29.045916 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.045884    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"]
Sep 18 14:16:29.166050 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.166002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e695176-aecc-4c4a-85c9-014071f7ae2a-tls-certs\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.166050 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.166055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-model-cache\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.166400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.166163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-dshm\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.166400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.166340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-tmp-dir\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.166400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.166377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-home\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.166539 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.166404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcdcp\" (UniqueName: \"kubernetes.io/projected/3e695176-aecc-4c4a-85c9-014071f7ae2a-kube-api-access-qcdcp\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.267863 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.267804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-model-cache\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.268053 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.267877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-dshm\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.268053 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.268006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-tmp-dir\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.268053 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.268033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-home\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.268278 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.268060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcdcp\" (UniqueName: \"kubernetes.io/projected/3e695176-aecc-4c4a-85c9-014071f7ae2a-kube-api-access-qcdcp\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.268278 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.268117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e695176-aecc-4c4a-85c9-014071f7ae2a-tls-certs\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.268382 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.268302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-model-cache\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.268465 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.268437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-home\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.268525 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.268464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-tmp-dir\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.270352 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.270328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-dshm\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.270836 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.270815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e695176-aecc-4c4a-85c9-014071f7ae2a-tls-certs\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.281139 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.281054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcdcp\" (UniqueName: \"kubernetes.io/projected/3e695176-aecc-4c4a-85c9-014071f7ae2a-kube-api-access-qcdcp\") pod \"noten-model-e8ad4e-kserve-dd6564c95-kz8gs\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") " pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.343222 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.343170    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:29.632340 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:29.632308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"]
Sep 18 14:16:29.632773 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:16:29.632744    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e695176_aecc_4c4a_85c9_014071f7ae2a.slice/crio-238c5d2f0505bd5b24ae8e5a1dd93c269808cc35c15988566828d3094bd01d8b WatchSource:0}: Error finding container 238c5d2f0505bd5b24ae8e5a1dd93c269808cc35c15988566828d3094bd01d8b: Status 404 returned error can't find the container with id 238c5d2f0505bd5b24ae8e5a1dd93c269808cc35c15988566828d3094bd01d8b
Sep 18 14:16:30.427197 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:30.427122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs" event={"ID":"3e695176-aecc-4c4a-85c9-014071f7ae2a","Type":"ContainerStarted","Data":"aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20"}
Sep 18 14:16:30.427658 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:30.427209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs" event={"ID":"3e695176-aecc-4c4a-85c9-014071f7ae2a","Type":"ContainerStarted","Data":"238c5d2f0505bd5b24ae8e5a1dd93c269808cc35c15988566828d3094bd01d8b"}
Sep 18 14:16:30.451353 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:30.451291    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs" podStartSLOduration=1.451269735 podStartE2EDuration="1.451269735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:16:30.449279898 +0000 UTC m=+1078.605800171" watchObservedRunningTime="2026-09-18 14:16:30.451269735 +0000 UTC m=+1078.607790016"
Sep 18 14:16:39.343824 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:39.343779    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:39.344410 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:39.343857    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:39.357048 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:39.357013    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:39.503538 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:39.503504    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:16:45.690311 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.690270    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"]
Sep 18 14:16:45.695607 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.695579    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.698375 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.698348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-93827d\"/\"e2e-conflict-b-1b2f9999-kserve-self-signed-certs\""
Sep 18 14:16:45.720461 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.720418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"]
Sep 18 14:16:45.846665 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.846619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-model-cache\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.846874 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.846684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-home\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.846874 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.846749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tmp-dir\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.846874 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.846809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q4nr\" (UniqueName: \"kubernetes.io/projected/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-kube-api-access-6q4nr\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.846874 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.846841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-dshm\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.847036 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.846879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tls-certs\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.948347 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.948235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-home\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.948347 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.948283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tmp-dir\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.948347 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.948340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q4nr\" (UniqueName: \"kubernetes.io/projected/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-kube-api-access-6q4nr\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.948652 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.948373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-dshm\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.948652 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.948427    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tls-certs\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.948652 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.948519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-model-cache\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.948819 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.948699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-home\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.948819 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.948771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tmp-dir\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.948934 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.948922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-model-cache\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.951017 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.950988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-dshm\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.951296 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.951275    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tls-certs\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:45.957296 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:45.957266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q4nr\" (UniqueName: \"kubernetes.io/projected/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-kube-api-access-6q4nr\") pod \"e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") " pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:46.005096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:46.005047    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:46.290073 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:16:46.290036    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2ae6c33_36dd_4e24_aa06_32a79bf567ec.slice/crio-eaa581f9749d95a10f54e8b0ab4044d6ea464eaf51c59b94bb1916cff976c4c5 WatchSource:0}: Error finding container eaa581f9749d95a10f54e8b0ab4044d6ea464eaf51c59b94bb1916cff976c4c5: Status 404 returned error can't find the container with id eaa581f9749d95a10f54e8b0ab4044d6ea464eaf51c59b94bb1916cff976c4c5
Sep 18 14:16:46.292264 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:46.292236    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"]
Sep 18 14:16:46.538741 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:46.538631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh" event={"ID":"b2ae6c33-36dd-4e24-aa06-32a79bf567ec","Type":"ContainerStarted","Data":"82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8"}
Sep 18 14:16:46.538741 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:46.538687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh" event={"ID":"b2ae6c33-36dd-4e24-aa06-32a79bf567ec","Type":"ContainerStarted","Data":"eaa581f9749d95a10f54e8b0ab4044d6ea464eaf51c59b94bb1916cff976c4c5"}
Sep 18 14:16:46.558258 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:46.558194    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh" podStartSLOduration=1.558175278 podStartE2EDuration="1.558175278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:16:46.556893385 +0000 UTC m=+1094.713413665" watchObservedRunningTime="2026-09-18 14:16:46.558175278 +0000 UTC m=+1094.714695558"
Sep 18 14:16:48.932827 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:48.932778    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:16:56.005484 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:56.005443    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:56.005920 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:56.005542    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:56.018774 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:56.018742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:56.618517 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:56.618484    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:16:57.198818 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:57.198774    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"]
Sep 18 14:16:57.312791 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:57.312754    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"]
Sep 18 14:16:57.313064 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:57.313020    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk" podUID="c18484eb-902e-4e1c-8eb4-a622e13a3110" containerName="istio-proxy" containerID="cri-o://1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70" gracePeriod=30
Sep 18 14:16:59.375110 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:16:59.375054    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"]
Sep 18 14:17:02.580535 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.580403    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:17:02.647173 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.647116    2563 generic.go:356] "Generic (PLEG): container finished" podID="c18484eb-902e-4e1c-8eb4-a622e13a3110" containerID="1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70" exitCode=0
Sep 18 14:17:02.647372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.647170    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk" event={"ID":"c18484eb-902e-4e1c-8eb4-a622e13a3110","Type":"ContainerDied","Data":"1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70"}
Sep 18 14:17:02.647372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.647220    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"
Sep 18 14:17:02.647372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.647235    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk" event={"ID":"c18484eb-902e-4e1c-8eb4-a622e13a3110","Type":"ContainerDied","Data":"486e3ba2fba17943bf34f2ea4948e18087381e11543981e221382d1558a85af5"}
Sep 18 14:17:02.647372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.647263    2563 scope.go:117] "RemoveContainer" containerID="1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70"
Sep 18 14:17:02.658708 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.658684    2563 scope.go:117] "RemoveContainer" containerID="1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70"
Sep 18 14:17:02.659101 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:17:02.659081    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70\": container with ID starting with 1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70 not found: ID does not exist" containerID="1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70"
Sep 18 14:17:02.659178 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.659115    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70"} err="failed to get container status \"1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70\": rpc error: code = NotFound desc = could not find container \"1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70\": container with ID starting with 1cfd8beb279883170464b79a751430cfb21d8d0af6df5e2b0a5dbc134c4c8c70 not found: ID does not exist"
Sep 18 14:17:02.728477 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.728400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-data\") pod \"c18484eb-902e-4e1c-8eb4-a622e13a3110\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") "
Sep 18 14:17:02.728797 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.728523    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-envoy\") pod \"c18484eb-902e-4e1c-8eb4-a622e13a3110\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") "
Sep 18 14:17:02.728797 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.728598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-token\") pod \"c18484eb-902e-4e1c-8eb4-a622e13a3110\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") "
Sep 18 14:17:02.728797 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.728654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-socket\") pod \"c18484eb-902e-4e1c-8eb4-a622e13a3110\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") "
Sep 18 14:17:02.728797 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.728743    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-podinfo\") pod \"c18484eb-902e-4e1c-8eb4-a622e13a3110\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") "
Sep 18 14:17:02.728797 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.728782    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18484eb-902e-4e1c-8eb4-a622e13a3110-istiod-ca-cert\") pod \"c18484eb-902e-4e1c-8eb4-a622e13a3110\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") "
Sep 18 14:17:02.729065 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.728823    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-credential-socket\") pod \"c18484eb-902e-4e1c-8eb4-a622e13a3110\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") "
Sep 18 14:17:02.729065 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.728863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-certs\") pod \"c18484eb-902e-4e1c-8eb4-a622e13a3110\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") "
Sep 18 14:17:02.729065 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.728862    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-data" (OuterVolumeSpecName: "istio-data") pod "c18484eb-902e-4e1c-8eb4-a622e13a3110" (UID: "c18484eb-902e-4e1c-8eb4-a622e13a3110"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:02.729065 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.728917    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdvfg\" (UniqueName: \"kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-kube-api-access-pdvfg\") pod \"c18484eb-902e-4e1c-8eb4-a622e13a3110\" (UID: \"c18484eb-902e-4e1c-8eb4-a622e13a3110\") "
Sep 18 14:17:02.729327 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.729047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c18484eb-902e-4e1c-8eb4-a622e13a3110" (UID: "c18484eb-902e-4e1c-8eb4-a622e13a3110"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:02.729327 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.729288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c18484eb-902e-4e1c-8eb4-a622e13a3110" (UID: "c18484eb-902e-4e1c-8eb4-a622e13a3110"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:02.729327 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.729320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18484eb-902e-4e1c-8eb4-a622e13a3110-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c18484eb-902e-4e1c-8eb4-a622e13a3110" (UID: "c18484eb-902e-4e1c-8eb4-a622e13a3110"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:17:02.729711 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.729564    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:02.729711 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.729676    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c18484eb-902e-4e1c-8eb4-a622e13a3110-istiod-ca-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:02.730103 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.729653    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c18484eb-902e-4e1c-8eb4-a622e13a3110" (UID: "c18484eb-902e-4e1c-8eb4-a622e13a3110"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:02.730337 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.729690    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-workload-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:02.730651 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.730633    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-data\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:02.732562 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.732225    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c18484eb-902e-4e1c-8eb4-a622e13a3110" (UID: "c18484eb-902e-4e1c-8eb4-a622e13a3110"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:17:02.732562 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.732281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-token" (OuterVolumeSpecName: "istio-token") pod "c18484eb-902e-4e1c-8eb4-a622e13a3110" (UID: "c18484eb-902e-4e1c-8eb4-a622e13a3110"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:02.732948 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.732925    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-kube-api-access-pdvfg" (OuterVolumeSpecName: "kube-api-access-pdvfg") pod "c18484eb-902e-4e1c-8eb4-a622e13a3110" (UID: "c18484eb-902e-4e1c-8eb4-a622e13a3110"). InnerVolumeSpecName "kube-api-access-pdvfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:02.745238 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.745099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c18484eb-902e-4e1c-8eb4-a622e13a3110" (UID: "c18484eb-902e-4e1c-8eb4-a622e13a3110"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:02.831729 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.831609    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-envoy\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:02.831729 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.831654    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-token\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:02.831729 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.831667    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c18484eb-902e-4e1c-8eb4-a622e13a3110-istio-podinfo\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:02.831729 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.831679    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c18484eb-902e-4e1c-8eb4-a622e13a3110-credential-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:02.831729 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:02.831693    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdvfg\" (UniqueName: \"kubernetes.io/projected/c18484eb-902e-4e1c-8eb4-a622e13a3110-kube-api-access-pdvfg\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:03.011716 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:03.011666    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"]
Sep 18 14:17:03.018907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:03.018869    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-e8ad4e-openshift-default-f567666bb-w66gk"]
Sep 18 14:17:04.288796 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:04.288756    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c18484eb-902e-4e1c-8eb4-a622e13a3110" path="/var/lib/kubelet/pods/c18484eb-902e-4e1c-8eb4-a622e13a3110/volumes"
Sep 18 14:17:12.219150 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.219088    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs" podUID="3e695176-aecc-4c4a-85c9-014071f7ae2a" containerName="main" containerID="cri-o://aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20" gracePeriod=45
Sep 18 14:17:12.512727 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.512689    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:17:12.634868 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.634810    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-model-cache\") pod \"3e695176-aecc-4c4a-85c9-014071f7ae2a\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") "
Sep 18 14:17:12.635083 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.634896    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-dshm\") pod \"3e695176-aecc-4c4a-85c9-014071f7ae2a\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") "
Sep 18 14:17:12.635083 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.634954    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcdcp\" (UniqueName: \"kubernetes.io/projected/3e695176-aecc-4c4a-85c9-014071f7ae2a-kube-api-access-qcdcp\") pod \"3e695176-aecc-4c4a-85c9-014071f7ae2a\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") "
Sep 18 14:17:12.635083 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.635010    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e695176-aecc-4c4a-85c9-014071f7ae2a-tls-certs\") pod \"3e695176-aecc-4c4a-85c9-014071f7ae2a\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") "
Sep 18 14:17:12.635083 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.635068    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-tmp-dir\") pod \"3e695176-aecc-4c4a-85c9-014071f7ae2a\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") "
Sep 18 14:17:12.635372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.635167    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-home\") pod \"3e695176-aecc-4c4a-85c9-014071f7ae2a\" (UID: \"3e695176-aecc-4c4a-85c9-014071f7ae2a\") "
Sep 18 14:17:12.635372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.635224    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e695176-aecc-4c4a-85c9-014071f7ae2a" (UID: "3e695176-aecc-4c4a-85c9-014071f7ae2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:12.635484 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.635465    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e695176-aecc-4c4a-85c9-014071f7ae2a" (UID: "3e695176-aecc-4c4a-85c9-014071f7ae2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:12.635552 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.635482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-home" (OuterVolumeSpecName: "home") pod "3e695176-aecc-4c4a-85c9-014071f7ae2a" (UID: "3e695176-aecc-4c4a-85c9-014071f7ae2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:12.635658 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.635640    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:12.635699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.635659    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:12.635699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.635674    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:12.637489 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.637466    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-dshm" (OuterVolumeSpecName: "dshm") pod "3e695176-aecc-4c4a-85c9-014071f7ae2a" (UID: "3e695176-aecc-4c4a-85c9-014071f7ae2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:12.637938 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.637920    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e695176-aecc-4c4a-85c9-014071f7ae2a-kube-api-access-qcdcp" (OuterVolumeSpecName: "kube-api-access-qcdcp") pod "3e695176-aecc-4c4a-85c9-014071f7ae2a" (UID: "3e695176-aecc-4c4a-85c9-014071f7ae2a"). InnerVolumeSpecName "kube-api-access-qcdcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:12.638018 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.637932    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e695176-aecc-4c4a-85c9-014071f7ae2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e695176-aecc-4c4a-85c9-014071f7ae2a" (UID: "3e695176-aecc-4c4a-85c9-014071f7ae2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:12.720875 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.720836    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e695176-aecc-4c4a-85c9-014071f7ae2a" containerID="aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20" exitCode=0
Sep 18 14:17:12.721097 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.720893    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"
Sep 18 14:17:12.721097 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.720922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs" event={"ID":"3e695176-aecc-4c4a-85c9-014071f7ae2a","Type":"ContainerDied","Data":"aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20"}
Sep 18 14:17:12.721097 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.720975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs" event={"ID":"3e695176-aecc-4c4a-85c9-014071f7ae2a","Type":"ContainerDied","Data":"238c5d2f0505bd5b24ae8e5a1dd93c269808cc35c15988566828d3094bd01d8b"}
Sep 18 14:17:12.721097 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.720994    2563 scope.go:117] "RemoveContainer" containerID="aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20"
Sep 18 14:17:12.733257 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.733233    2563 scope.go:117] "RemoveContainer" containerID="aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20"
Sep 18 14:17:12.733635 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:17:12.733614    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20\": container with ID starting with aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20 not found: ID does not exist" containerID="aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20"
Sep 18 14:17:12.733719 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.733646    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20"} err="failed to get container status \"aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20\": rpc error: code = NotFound desc = could not find container \"aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20\": container with ID starting with aff3b1b64c0317193d590dfc5553233bb2c4b31a09798a11c78963eca6647c20 not found: ID does not exist"
Sep 18 14:17:12.736718 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.736692    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e695176-aecc-4c4a-85c9-014071f7ae2a-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:12.736718 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.736719    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcdcp\" (UniqueName: \"kubernetes.io/projected/3e695176-aecc-4c4a-85c9-014071f7ae2a-kube-api-access-qcdcp\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:12.736838 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.736730    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e695176-aecc-4c4a-85c9-014071f7ae2a-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:12.750532 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.750476    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"]
Sep 18 14:17:12.753458 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:12.753423    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-e8ad4e/noten-model-e8ad4e-kserve-dd6564c95-kz8gs"]
Sep 18 14:17:14.290770 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.290726    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e695176-aecc-4c4a-85c9-014071f7ae2a" path="/var/lib/kubelet/pods/3e695176-aecc-4c4a-85c9-014071f7ae2a/volumes"
Sep 18 14:17:14.395545 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.395500    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p" podUID="79c42fd9-e4e2-4c36-996f-78a964693ebe" containerName="main" containerID="cri-o://ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5" gracePeriod=45
Sep 18 14:17:14.672742 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.672705    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:17:14.738778 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.738736    2563 generic.go:356] "Generic (PLEG): container finished" podID="79c42fd9-e4e2-4c36-996f-78a964693ebe" containerID="ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5" exitCode=0
Sep 18 14:17:14.738991 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.738786    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"
Sep 18 14:17:14.738991 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.738824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p" event={"ID":"79c42fd9-e4e2-4c36-996f-78a964693ebe","Type":"ContainerDied","Data":"ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5"}
Sep 18 14:17:14.738991 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.738873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p" event={"ID":"79c42fd9-e4e2-4c36-996f-78a964693ebe","Type":"ContainerDied","Data":"148e696bfabebef2626883b67eb69e145d4e008ac8f1b688117668c739293b08"}
Sep 18 14:17:14.738991 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.738895    2563 scope.go:117] "RemoveContainer" containerID="ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5"
Sep 18 14:17:14.750250 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.750218    2563 scope.go:117] "RemoveContainer" containerID="ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5"
Sep 18 14:17:14.750683 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:17:14.750657    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5\": container with ID starting with ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5 not found: ID does not exist" containerID="ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5"
Sep 18 14:17:14.750807 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.750691    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5"} err="failed to get container status \"ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5\": rpc error: code = NotFound desc = could not find container \"ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5\": container with ID starting with ca675b5c85105eb162d65bca856eceeb33ea385830e2d0ef511e46dff10bf6e5 not found: ID does not exist"
Sep 18 14:17:14.760093 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.760047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c42fd9-e4e2-4c36-996f-78a964693ebe-tls-certs\") pod \"79c42fd9-e4e2-4c36-996f-78a964693ebe\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") "
Sep 18 14:17:14.760326 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.760118    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-dshm\") pod \"79c42fd9-e4e2-4c36-996f-78a964693ebe\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") "
Sep 18 14:17:14.760326 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.760230    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-home\") pod \"79c42fd9-e4e2-4c36-996f-78a964693ebe\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") "
Sep 18 14:17:14.760326 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.760282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-tmp-dir\") pod \"79c42fd9-e4e2-4c36-996f-78a964693ebe\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") "
Sep 18 14:17:14.760508 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.760356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpq5z\" (UniqueName: \"kubernetes.io/projected/79c42fd9-e4e2-4c36-996f-78a964693ebe-kube-api-access-tpq5z\") pod \"79c42fd9-e4e2-4c36-996f-78a964693ebe\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") "
Sep 18 14:17:14.760508 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.760390    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-model-cache\") pod \"79c42fd9-e4e2-4c36-996f-78a964693ebe\" (UID: \"79c42fd9-e4e2-4c36-996f-78a964693ebe\") "
Sep 18 14:17:14.760623 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.760539    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-home" (OuterVolumeSpecName: "home") pod "79c42fd9-e4e2-4c36-996f-78a964693ebe" (UID: "79c42fd9-e4e2-4c36-996f-78a964693ebe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:14.760682 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.760614    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "79c42fd9-e4e2-4c36-996f-78a964693ebe" (UID: "79c42fd9-e4e2-4c36-996f-78a964693ebe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:14.760888 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.760840    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:14.760888 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.760862    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:14.760888 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.760877    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-model-cache" (OuterVolumeSpecName: "model-cache") pod "79c42fd9-e4e2-4c36-996f-78a964693ebe" (UID: "79c42fd9-e4e2-4c36-996f-78a964693ebe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:14.763210 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.763180    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c42fd9-e4e2-4c36-996f-78a964693ebe-kube-api-access-tpq5z" (OuterVolumeSpecName: "kube-api-access-tpq5z") pod "79c42fd9-e4e2-4c36-996f-78a964693ebe" (UID: "79c42fd9-e4e2-4c36-996f-78a964693ebe"). InnerVolumeSpecName "kube-api-access-tpq5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:14.763359 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.763273    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-dshm" (OuterVolumeSpecName: "dshm") pod "79c42fd9-e4e2-4c36-996f-78a964693ebe" (UID: "79c42fd9-e4e2-4c36-996f-78a964693ebe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:14.763359 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.763300    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c42fd9-e4e2-4c36-996f-78a964693ebe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79c42fd9-e4e2-4c36-996f-78a964693ebe" (UID: "79c42fd9-e4e2-4c36-996f-78a964693ebe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:14.862243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.862121    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79c42fd9-e4e2-4c36-996f-78a964693ebe-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:14.862243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.862177    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:14.862243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.862188    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpq5z\" (UniqueName: \"kubernetes.io/projected/79c42fd9-e4e2-4c36-996f-78a964693ebe-kube-api-access-tpq5z\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:14.862243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:14.862197    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79c42fd9-e4e2-4c36-996f-78a964693ebe-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:15.065524 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:15.065484    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"]
Sep 18 14:17:15.069505 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:15.069464    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-dc2cee/test-model-dc2cee-kserve-668594485c-97s8p"]
Sep 18 14:17:16.289305 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:16.289256    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c42fd9-e4e2-4c36-996f-78a964693ebe" path="/var/lib/kubelet/pods/79c42fd9-e4e2-4c36-996f-78a964693ebe/volumes"
Sep 18 14:17:21.456415 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.456363    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"]
Sep 18 14:17:21.456957 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.456887    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c42fd9-e4e2-4c36-996f-78a964693ebe" containerName="main"
Sep 18 14:17:21.456957 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.456904    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c42fd9-e4e2-4c36-996f-78a964693ebe" containerName="main"
Sep 18 14:17:21.456957 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.456921    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e695176-aecc-4c4a-85c9-014071f7ae2a" containerName="main"
Sep 18 14:17:21.456957 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.456927    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e695176-aecc-4c4a-85c9-014071f7ae2a" containerName="main"
Sep 18 14:17:21.456957 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.456936    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c18484eb-902e-4e1c-8eb4-a622e13a3110" containerName="istio-proxy"
Sep 18 14:17:21.456957 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.456942    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c18484eb-902e-4e1c-8eb4-a622e13a3110" containerName="istio-proxy"
Sep 18 14:17:21.457197 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.457046    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c18484eb-902e-4e1c-8eb4-a622e13a3110" containerName="istio-proxy"
Sep 18 14:17:21.457197 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.457055    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c42fd9-e4e2-4c36-996f-78a964693ebe" containerName="main"
Sep 18 14:17:21.457197 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.457065    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e695176-aecc-4c4a-85c9-014071f7ae2a" containerName="main"
Sep 18 14:17:21.462162 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.462117    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:21.467124 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.467093    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 14:17:21.467714 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.467688    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-526b97-serving-cert\""
Sep 18 14:17:21.467886 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.467730    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5nh5f\""
Sep 18 14:17:21.467886 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.467818    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 14:17:21.467886 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.467696    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 14:17:21.478383 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.478344    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"]
Sep 18 14:17:21.530183 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.530105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:21.530183 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.530178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:21.530515 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.530211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hkgw\" (UniqueName: \"kubernetes.io/projected/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-kube-api-access-6hkgw\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:21.530515 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.530304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:21.631693 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.631630    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:21.631693 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.631684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:21.631972 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.631713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hkgw\" (UniqueName: \"kubernetes.io/projected/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-kube-api-access-6hkgw\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:21.631972 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.631782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:21.631972 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:17:21.631836    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-serving-cert: secret "maas-api-e2e-shared-a-w2-526b97-serving-cert" not found
Sep 18 14:17:21.631972 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:17:21.631972    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls podName:153fae7e-a7a2-4bd5-9c89-5a1b405b6b64 nodeName:}" failed. No retries permitted until 2026-09-18 14:17:22.131945808 +0000 UTC m=+1130.288466101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls") pod "maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj" (UID: "153fae7e-a7a2-4bd5-9c89-5a1b405b6b64") : secret "maas-api-e2e-shared-a-w2-526b97-serving-cert" not found
Sep 18 14:17:21.632592 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.632567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:21.634881 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.634853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:21.648977 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:21.648947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hkgw\" (UniqueName: \"kubernetes.io/projected/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-kube-api-access-6hkgw\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:22.138189 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:22.138140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:22.138426 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:17:22.138252    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-serving-cert: secret "maas-api-e2e-shared-a-w2-526b97-serving-cert" not found
Sep 18 14:17:22.138426 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:17:22.138344    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls podName:153fae7e-a7a2-4bd5-9c89-5a1b405b6b64 nodeName:}" failed. No retries permitted until 2026-09-18 14:17:23.138325487 +0000 UTC m=+1131.294845746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls") pod "maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj" (UID: "153fae7e-a7a2-4bd5-9c89-5a1b405b6b64") : secret "maas-api-e2e-shared-a-w2-526b97-serving-cert" not found
Sep 18 14:17:22.725532 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:22.725479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:17:23.149626 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:23.149512    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:23.152352 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:23.152320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:23.280563 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:23.280515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:23.568717 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:23.568673    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"]
Sep 18 14:17:23.569796 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:17:23.569756    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153fae7e_a7a2_4bd5_9c89_5a1b405b6b64.slice/crio-3260d672fbaeb669917df1e4e9823a7f5313dd9d9321901471de26f19edc95c1 WatchSource:0}: Error finding container 3260d672fbaeb669917df1e4e9823a7f5313dd9d9321901471de26f19edc95c1: Status 404 returned error can't find the container with id 3260d672fbaeb669917df1e4e9823a7f5313dd9d9321901471de26f19edc95c1
Sep 18 14:17:23.571626 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:23.571583    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:17:23.805618 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:23.805571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj" event={"ID":"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64","Type":"ContainerStarted","Data":"3260d672fbaeb669917df1e4e9823a7f5313dd9d9321901471de26f19edc95c1"}
Sep 18 14:17:24.814211 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:24.814163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj" event={"ID":"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64","Type":"ContainerStarted","Data":"55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187"}
Sep 18 14:17:24.814690 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:24.814271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:24.836089 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:24.836028    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj" podStartSLOduration=3.425776464 podStartE2EDuration="3.836005491s" totalImagesPullingTime="410.229027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:17:24.833575066 +0000 UTC m=+1132.990095359" watchObservedRunningTime="2026-09-18 14:17:24.836005491 +0000 UTC m=+1132.992525771"
Sep 18 14:17:25.148378 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:25.148249    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"]
Sep 18 14:17:29.601510 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:29.601465    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"]
Sep 18 14:17:30.827622 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:30.827577    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:17:40.170691 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.170627    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh" podUID="b2ae6c33-36dd-4e24-aa06-32a79bf567ec" containerName="main" containerID="cri-o://82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8" gracePeriod=45
Sep 18 14:17:40.442217 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.442178    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:17:40.546818 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.546773    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-model-cache\") pod \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") "
Sep 18 14:17:40.546818 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.546821    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tmp-dir\") pod \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") "
Sep 18 14:17:40.547123 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.546847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-dshm\") pod \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") "
Sep 18 14:17:40.547123 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.546877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q4nr\" (UniqueName: \"kubernetes.io/projected/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-kube-api-access-6q4nr\") pod \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") "
Sep 18 14:17:40.547123 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.546905    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-home\") pod \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") "
Sep 18 14:17:40.547123 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.546943    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tls-certs\") pod \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\" (UID: \"b2ae6c33-36dd-4e24-aa06-32a79bf567ec\") "
Sep 18 14:17:40.547385 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.547208    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2ae6c33-36dd-4e24-aa06-32a79bf567ec" (UID: "b2ae6c33-36dd-4e24-aa06-32a79bf567ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:40.547385 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.547225    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2ae6c33-36dd-4e24-aa06-32a79bf567ec" (UID: "b2ae6c33-36dd-4e24-aa06-32a79bf567ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:40.547385 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.547308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-home" (OuterVolumeSpecName: "home") pod "b2ae6c33-36dd-4e24-aa06-32a79bf567ec" (UID: "b2ae6c33-36dd-4e24-aa06-32a79bf567ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:40.547385 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.547376    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:40.547530 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.547395    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:40.549643 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.549604    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-dshm" (OuterVolumeSpecName: "dshm") pod "b2ae6c33-36dd-4e24-aa06-32a79bf567ec" (UID: "b2ae6c33-36dd-4e24-aa06-32a79bf567ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:40.549775 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.549667    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-kube-api-access-6q4nr" (OuterVolumeSpecName: "kube-api-access-6q4nr") pod "b2ae6c33-36dd-4e24-aa06-32a79bf567ec" (UID: "b2ae6c33-36dd-4e24-aa06-32a79bf567ec"). InnerVolumeSpecName "kube-api-access-6q4nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:40.549775 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.549669    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2ae6c33-36dd-4e24-aa06-32a79bf567ec" (UID: "b2ae6c33-36dd-4e24-aa06-32a79bf567ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:40.648511 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.648465    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q4nr\" (UniqueName: \"kubernetes.io/projected/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-kube-api-access-6q4nr\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:40.648511 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.648516    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:40.648769 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.648527    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:40.648769 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.648540    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ae6c33-36dd-4e24-aa06-32a79bf567ec-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:40.932735 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.932689    2563 generic.go:356] "Generic (PLEG): container finished" podID="b2ae6c33-36dd-4e24-aa06-32a79bf567ec" containerID="82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8" exitCode=0
Sep 18 14:17:40.932960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.932741    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"
Sep 18 14:17:40.932960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.932767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh" event={"ID":"b2ae6c33-36dd-4e24-aa06-32a79bf567ec","Type":"ContainerDied","Data":"82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8"}
Sep 18 14:17:40.932960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.932808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh" event={"ID":"b2ae6c33-36dd-4e24-aa06-32a79bf567ec","Type":"ContainerDied","Data":"eaa581f9749d95a10f54e8b0ab4044d6ea464eaf51c59b94bb1916cff976c4c5"}
Sep 18 14:17:40.932960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.932829    2563 scope.go:117] "RemoveContainer" containerID="82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8"
Sep 18 14:17:40.944923 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.944893    2563 scope.go:117] "RemoveContainer" containerID="82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8"
Sep 18 14:17:40.945325 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:17:40.945301    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8\": container with ID starting with 82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8 not found: ID does not exist" containerID="82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8"
Sep 18 14:17:40.945430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.945337    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8"} err="failed to get container status \"82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8\": rpc error: code = NotFound desc = could not find container \"82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8\": container with ID starting with 82f2e36df5f22758aeff8681da9dc85d136e01bc22bebc8bab1953e0e55674d8 not found: ID does not exist"
Sep 18 14:17:40.960051 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.960007    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"]
Sep 18 14:17:40.964731 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:40.964687    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-b-1b2f9999-kserve-5ff6cb5574-8zlgh"]
Sep 18 14:17:42.290944 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:42.290902    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2ae6c33-36dd-4e24-aa06-32a79bf567ec" path="/var/lib/kubelet/pods/b2ae6c33-36dd-4e24-aa06-32a79bf567ec/volumes"
Sep 18 14:17:44.623177 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.623076    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq" podUID="e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" containerName="main" containerID="cri-o://be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0" gracePeriod=45
Sep 18 14:17:44.891455 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.891368    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:17:44.962882 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.962843    2563 generic.go:356] "Generic (PLEG): container finished" podID="e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" containerID="be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0" exitCode=0
Sep 18 14:17:44.963124 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.962892    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"
Sep 18 14:17:44.963124 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.962921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq" event={"ID":"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a","Type":"ContainerDied","Data":"be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0"}
Sep 18 14:17:44.963124 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.962978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq" event={"ID":"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a","Type":"ContainerDied","Data":"1a3e71270239ec917b5703f869853d6c7e4c76d71555198c5bc4001b7316f053"}
Sep 18 14:17:44.963124 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.962999    2563 scope.go:117] "RemoveContainer" containerID="be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0"
Sep 18 14:17:44.975027 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.975004    2563 scope.go:117] "RemoveContainer" containerID="be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0"
Sep 18 14:17:44.975413 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:17:44.975392    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0\": container with ID starting with be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0 not found: ID does not exist" containerID="be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0"
Sep 18 14:17:44.975503 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.975424    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0"} err="failed to get container status \"be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0\": rpc error: code = NotFound desc = could not find container \"be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0\": container with ID starting with be85dfcdd9d11d5d8df4f154944b82cb19b715fc84c279c8340babe700788fd0 not found: ID does not exist"
Sep 18 14:17:44.999960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.999924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-dshm\") pod \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") "
Sep 18 14:17:44.999960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.999959    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-model-cache\") pod \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") "
Sep 18 14:17:45.000235 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:44.999984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-home\") pod \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") "
Sep 18 14:17:45.000235 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.000033    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mf6z\" (UniqueName: \"kubernetes.io/projected/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-kube-api-access-2mf6z\") pod \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") "
Sep 18 14:17:45.000235 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.000200    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tmp-dir\") pod \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") "
Sep 18 14:17:45.000400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.000257    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tls-certs\") pod \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\" (UID: \"e06b7348-4fa2-4ea5-ae0e-6b5dd621943a\") "
Sep 18 14:17:45.000400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.000268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-home" (OuterVolumeSpecName: "home") pod "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" (UID: "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:45.000400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.000308    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" (UID: "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:45.000556 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.000479    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" (UID: "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:45.000687 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.000658    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:45.000687 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.000681    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:45.000687 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.000692    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:45.002508 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.002478    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" (UID: "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:17:45.002780 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.002765    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-dshm" (OuterVolumeSpecName: "dshm") pod "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" (UID: "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:17:45.002850 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.002806    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-kube-api-access-2mf6z" (OuterVolumeSpecName: "kube-api-access-2mf6z") pod "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" (UID: "e06b7348-4fa2-4ea5-ae0e-6b5dd621943a"). InnerVolumeSpecName "kube-api-access-2mf6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:45.101673 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.101619    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:45.101673 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.101661    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:45.101673 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.101673    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mf6z\" (UniqueName: \"kubernetes.io/projected/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a-kube-api-access-2mf6z\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:45.296336 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.296291    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"]
Sep 18 14:17:45.299172 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:45.299145    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/e2e-conflict-a-1b2f9999-kserve-7549bf688d-vtlpq"]
Sep 18 14:17:46.295551 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:46.291005    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" path="/var/lib/kubelet/pods/e06b7348-4fa2-4ea5-ae0e-6b5dd621943a/volumes"
Sep 18 14:17:51.072003 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:51.071962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:17:56.132689 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.132582    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"]
Sep 18 14:17:56.133339 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.132818    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f" podUID="2387a594-7667-4484-9eb6-731d02d12e8c" containerName="payload-processing" containerID="cri-o://858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b" gracePeriod=30
Sep 18 14:17:56.142480 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.142434    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"]
Sep 18 14:17:56.143165 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.142677    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6" podUID="ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8" containerName="payload-pre-processing" containerID="cri-o://cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823" gracePeriod=30
Sep 18 14:17:56.439041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.439004    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:17:56.442061 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.442039    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:17:56.523413 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.523373    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfmqc\" (UniqueName: \"kubernetes.io/projected/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-kube-api-access-mfmqc\") pod \"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8\" (UID: \"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8\") "
Sep 18 14:17:56.523642 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.523449    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-plugins-config-volume\") pod \"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8\" (UID: \"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8\") "
Sep 18 14:17:56.523642 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.523487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2387a594-7667-4484-9eb6-731d02d12e8c-plugins-config-volume\") pod \"2387a594-7667-4484-9eb6-731d02d12e8c\" (UID: \"2387a594-7667-4484-9eb6-731d02d12e8c\") "
Sep 18 14:17:56.523642 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.523510    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dqb9\" (UniqueName: \"kubernetes.io/projected/2387a594-7667-4484-9eb6-731d02d12e8c-kube-api-access-2dqb9\") pod \"2387a594-7667-4484-9eb6-731d02d12e8c\" (UID: \"2387a594-7667-4484-9eb6-731d02d12e8c\") "
Sep 18 14:17:56.523951 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.523921    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2387a594-7667-4484-9eb6-731d02d12e8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2387a594-7667-4484-9eb6-731d02d12e8c" (UID: "2387a594-7667-4484-9eb6-731d02d12e8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:17:56.524044 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.523922    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8" (UID: "ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:17:56.525869 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.525836    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2387a594-7667-4484-9eb6-731d02d12e8c-kube-api-access-2dqb9" (OuterVolumeSpecName: "kube-api-access-2dqb9") pod "2387a594-7667-4484-9eb6-731d02d12e8c" (UID: "2387a594-7667-4484-9eb6-731d02d12e8c"). InnerVolumeSpecName "kube-api-access-2dqb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:56.526000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.525875    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-kube-api-access-mfmqc" (OuterVolumeSpecName: "kube-api-access-mfmqc") pod "ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8" (UID: "ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8"). InnerVolumeSpecName "kube-api-access-mfmqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:17:56.625249 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.625207    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-plugins-config-volume\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:56.625249 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.625242    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2387a594-7667-4484-9eb6-731d02d12e8c-plugins-config-volume\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:56.625249 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.625254    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dqb9\" (UniqueName: \"kubernetes.io/projected/2387a594-7667-4484-9eb6-731d02d12e8c-kube-api-access-2dqb9\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:56.625249 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:56.625263    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfmqc\" (UniqueName: \"kubernetes.io/projected/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8-kube-api-access-mfmqc\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:17:57.043582 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.043541    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8" containerID="cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823" exitCode=0
Sep 18 14:17:57.043582 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.043597    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"
Sep 18 14:17:57.043905 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.043638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6" event={"ID":"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8","Type":"ContainerDied","Data":"cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823"}
Sep 18 14:17:57.043905 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.043710    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6" event={"ID":"ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8","Type":"ContainerDied","Data":"42b7aa0607e4b60726ef31bdd61aeae48420b7cf766331a1bc2626fa66e6b15a"}
Sep 18 14:17:57.043905 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.043740    2563 scope.go:117] "RemoveContainer" containerID="cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823"
Sep 18 14:17:57.045322 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.045293    2563 generic.go:356] "Generic (PLEG): container finished" podID="2387a594-7667-4484-9eb6-731d02d12e8c" containerID="858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b" exitCode=0
Sep 18 14:17:57.045448 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.045337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f" event={"ID":"2387a594-7667-4484-9eb6-731d02d12e8c","Type":"ContainerDied","Data":"858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b"}
Sep 18 14:17:57.045448 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.045401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f" event={"ID":"2387a594-7667-4484-9eb6-731d02d12e8c","Type":"ContainerDied","Data":"df310eecb6a85bfc32b8ce2e01709227d68771ef09814114b4bded1bb0d1db22"}
Sep 18 14:17:57.045448 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.045349    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"
Sep 18 14:17:57.055717 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.055695    2563 scope.go:117] "RemoveContainer" containerID="cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823"
Sep 18 14:17:57.056102 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:17:57.056070    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823\": container with ID starting with cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823 not found: ID does not exist" containerID="cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823"
Sep 18 14:17:57.056206 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.056115    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823"} err="failed to get container status \"cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823\": rpc error: code = NotFound desc = could not find container \"cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823\": container with ID starting with cc16e61a7389a1fcffed9ae101c81fd711331a58eb4068e16fc58abf7e8b6823 not found: ID does not exist"
Sep 18 14:17:57.056206 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.056162    2563 scope.go:117] "RemoveContainer" containerID="858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b"
Sep 18 14:17:57.069637 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.069596    2563 scope.go:117] "RemoveContainer" containerID="858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b"
Sep 18 14:17:57.070307 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:17:57.070266    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b\": container with ID starting with 858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b not found: ID does not exist" containerID="858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b"
Sep 18 14:17:57.070420 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.070312    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b"} err="failed to get container status \"858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b\": rpc error: code = NotFound desc = could not find container \"858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b\": container with ID starting with 858e0e55b5d6c553f6dcabd1b0065fd6898f6a7da8c27818d05da86fd115516b not found: ID does not exist"
Sep 18 14:17:57.071194 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.071170    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"]
Sep 18 14:17:57.075485 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.075460    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ee8a08-868557b686-h5jr6"]
Sep 18 14:17:57.085907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.085875    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"]
Sep 18 14:17:57.092368 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:57.092339    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ee8a08-8665f8d76f-h2p2f"]
Sep 18 14:17:58.289111 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:58.289068    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2387a594-7667-4484-9eb6-731d02d12e8c" path="/var/lib/kubelet/pods/2387a594-7667-4484-9eb6-731d02d12e8c/volumes"
Sep 18 14:17:58.289539 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:58.289506    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8" path="/var/lib/kubelet/pods/ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8/volumes"
Sep 18 14:17:59.806146 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806096    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"]
Sep 18 14:17:59.806565 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806528    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ae6c33-36dd-4e24-aa06-32a79bf567ec" containerName="main"
Sep 18 14:17:59.806565 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806543    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ae6c33-36dd-4e24-aa06-32a79bf567ec" containerName="main"
Sep 18 14:17:59.806565 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806555    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2387a594-7667-4484-9eb6-731d02d12e8c" containerName="payload-processing"
Sep 18 14:17:59.806565 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806561    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2387a594-7667-4484-9eb6-731d02d12e8c" containerName="payload-processing"
Sep 18 14:17:59.806701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806576    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" containerName="main"
Sep 18 14:17:59.806701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806581    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" containerName="main"
Sep 18 14:17:59.806701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806589    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8" containerName="payload-pre-processing"
Sep 18 14:17:59.806701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806594    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8" containerName="payload-pre-processing"
Sep 18 14:17:59.806701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806656    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2387a594-7667-4484-9eb6-731d02d12e8c" containerName="payload-processing"
Sep 18 14:17:59.806701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806666    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e06b7348-4fa2-4ea5-ae0e-6b5dd621943a" containerName="main"
Sep 18 14:17:59.806701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806672    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2ae6c33-36dd-4e24-aa06-32a79bf567ec" containerName="main"
Sep 18 14:17:59.806701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.806678    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce06b21a-2a31-4c70-ad4b-4cf9f92b4cf8" containerName="payload-pre-processing"
Sep 18 14:17:59.811552 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.811531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:17:59.814291 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.814262    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4d9222\"/\"distinct-2-w0-4d9222-kserve-self-signed-certs\""
Sep 18 14:17:59.826166 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.826118    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"]
Sep 18 14:17:59.960681 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.960639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-tmp-dir\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:17:59.960681 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.960680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-dshm\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:17:59.960923 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.960707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-model-cache\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:17:59.960923 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.960817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qprxw\" (UniqueName: \"kubernetes.io/projected/b3ac69e0-a9c9-4c14-a717-73854425340a-kube-api-access-qprxw\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:17:59.960923 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.960865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ac69e0-a9c9-4c14-a717-73854425340a-tls-certs\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:17:59.961029 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:17:59.960967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-home\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.062123 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.062016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ac69e0-a9c9-4c14-a717-73854425340a-tls-certs\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.062123 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.062108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-home\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.062400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.062187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-tmp-dir\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.062400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.062204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-dshm\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.062400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.062227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-model-cache\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.062400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.062318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprxw\" (UniqueName: \"kubernetes.io/projected/b3ac69e0-a9c9-4c14-a717-73854425340a-kube-api-access-qprxw\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.062613 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.062579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-home\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.062669 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.062644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-model-cache\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.062732 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.062671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-tmp-dir\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.065344 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.065313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ac69e0-a9c9-4c14-a717-73854425340a-tls-certs\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.065531 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.065371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-dshm\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.072733 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.072694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprxw\" (UniqueName: \"kubernetes.io/projected/b3ac69e0-a9c9-4c14-a717-73854425340a-kube-api-access-qprxw\") pod \"distinct-2-w0-4d9222-kserve-55584545f5-ngkjd\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") " pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.122048 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.122008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:00.402322 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:00.402281    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"]
Sep 18 14:18:00.403562 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:18:00.403529    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3ac69e0_a9c9_4c14_a717_73854425340a.slice/crio-4a06fc6cef7dc40a23fa20aa4a85e39bd8bb6d8a826a43a37a2a2b59d732155f WatchSource:0}: Error finding container 4a06fc6cef7dc40a23fa20aa4a85e39bd8bb6d8a826a43a37a2a2b59d732155f: Status 404 returned error can't find the container with id 4a06fc6cef7dc40a23fa20aa4a85e39bd8bb6d8a826a43a37a2a2b59d732155f
Sep 18 14:18:01.080701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:01.080658    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd" event={"ID":"b3ac69e0-a9c9-4c14-a717-73854425340a","Type":"ContainerStarted","Data":"b70dd65d2c9abfa6b00d4b215a3fe2e8409b774e41cc7516640802fe090abb35"}
Sep 18 14:18:01.081171 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:01.080709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd" event={"ID":"b3ac69e0-a9c9-4c14-a717-73854425340a","Type":"ContainerStarted","Data":"4a06fc6cef7dc40a23fa20aa4a85e39bd8bb6d8a826a43a37a2a2b59d732155f"}
Sep 18 14:18:01.103338 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:01.103280    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd" podStartSLOduration=2.103261996 podStartE2EDuration="2.103261996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:01.101409066 +0000 UTC m=+1169.257929349" watchObservedRunningTime="2026-09-18 14:18:01.103261996 +0000 UTC m=+1169.259782277"
Sep 18 14:18:02.445002 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:02.444957    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7758-7be963"]
Sep 18 14:18:02.448550 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:02.448528    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:02.451469 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:02.451443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r4cmx\""
Sep 18 14:18:02.456043 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:02.456011    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7758-7be963"]
Sep 18 14:18:02.590014 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:02.589973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r6q4\" (UniqueName: \"kubernetes.io/projected/eb0508fc-5d2c-4162-929e-d6b0e88b1866-kube-api-access-9r6q4\") pod \"test-curl-7758-7be963\" (UID: \"eb0508fc-5d2c-4162-929e-d6b0e88b1866\") " pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:02.691193 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:02.691110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r6q4\" (UniqueName: \"kubernetes.io/projected/eb0508fc-5d2c-4162-929e-d6b0e88b1866-kube-api-access-9r6q4\") pod \"test-curl-7758-7be963\" (UID: \"eb0508fc-5d2c-4162-929e-d6b0e88b1866\") " pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:02.700616 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:02.700532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r6q4\" (UniqueName: \"kubernetes.io/projected/eb0508fc-5d2c-4162-929e-d6b0e88b1866-kube-api-access-9r6q4\") pod \"test-curl-7758-7be963\" (UID: \"eb0508fc-5d2c-4162-929e-d6b0e88b1866\") " pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:02.761739 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:02.761686    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:03.048202 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:03.048167    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7758-7be963"]
Sep 18 14:18:03.049984 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:18:03.049950    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb0508fc_5d2c_4162_929e_d6b0e88b1866.slice/crio-e1d2282f8103ff659c7a2bc350939ba08389284c51bdc18a827e00c09d27e1a8 WatchSource:0}: Error finding container e1d2282f8103ff659c7a2bc350939ba08389284c51bdc18a827e00c09d27e1a8: Status 404 returned error can't find the container with id e1d2282f8103ff659c7a2bc350939ba08389284c51bdc18a827e00c09d27e1a8
Sep 18 14:18:03.096053 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:03.096017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7758-7be963" event={"ID":"eb0508fc-5d2c-4162-929e-d6b0e88b1866","Type":"ContainerStarted","Data":"e1d2282f8103ff659c7a2bc350939ba08389284c51bdc18a827e00c09d27e1a8"}
Sep 18 14:18:04.106230 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.106185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7758-7be963" event={"ID":"eb0508fc-5d2c-4162-929e-d6b0e88b1866","Type":"ContainerStarted","Data":"defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61"}
Sep 18 14:18:04.127796 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.127736    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7758-7be963" podStartSLOduration=2.127715213 podStartE2EDuration="2.127715213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:04.126676252 +0000 UTC m=+1172.283196533" watchObservedRunningTime="2026-09-18 14:18:04.127715213 +0000 UTC m=+1172.284235494"
Sep 18 14:18:04.358410 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.358317    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"]
Sep 18 14:18:04.358598 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.358571    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy" containerID="cri-o://ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c" gracePeriod=30
Sep 18 14:18:04.847343 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.847297    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7758-7be963"]
Sep 18 14:18:04.850646 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.850614    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7758-7be963"]
Sep 18 14:18:04.968819 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.968768    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7758-453c2e"]
Sep 18 14:18:04.969332 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.969309    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" containerName="test-curl-7758-7be963"
Sep 18 14:18:04.969332 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.969334    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" containerName="test-curl-7758-7be963"
Sep 18 14:18:04.969501 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.969462    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" containerName="test-curl-7758-7be963"
Sep 18 14:18:04.972883 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.972857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:04.982889 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:04.982802    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7758-453c2e"]
Sep 18 14:18:05.113441 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:05.113404    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7758-7be963" container="test-curl-7758-7be963"
Sep 18 14:18:05.117350 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:05.117304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jp9s\" (UniqueName: \"kubernetes.io/projected/42cf7841-19ba-4fb5-a15a-d26645e949df-kube-api-access-8jp9s\") pod \"test-curl-7758-453c2e\" (UID: \"42cf7841-19ba-4fb5-a15a-d26645e949df\") " pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:05.218595 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:05.218556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jp9s\" (UniqueName: \"kubernetes.io/projected/42cf7841-19ba-4fb5-a15a-d26645e949df-kube-api-access-8jp9s\") pod \"test-curl-7758-453c2e\" (UID: \"42cf7841-19ba-4fb5-a15a-d26645e949df\") " pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:05.252498 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:05.252395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jp9s\" (UniqueName: \"kubernetes.io/projected/42cf7841-19ba-4fb5-a15a-d26645e949df-kube-api-access-8jp9s\") pod \"test-curl-7758-453c2e\" (UID: \"42cf7841-19ba-4fb5-a15a-d26645e949df\") " pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:05.284374 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:05.284331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:05.576101 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:05.576068    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7758-453c2e"]
Sep 18 14:18:05.578211 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:18:05.578173    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42cf7841_19ba_4fb5_a15a_d26645e949df.slice/crio-f27e8cda154460769380cdd47108957ba452f940a60b9f4b02884945e187217f WatchSource:0}: Error finding container f27e8cda154460769380cdd47108957ba452f940a60b9f4b02884945e187217f: Status 404 returned error can't find the container with id f27e8cda154460769380cdd47108957ba452f940a60b9f4b02884945e187217f
Sep 18 14:18:06.121872 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.121824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7758-453c2e" event={"ID":"42cf7841-19ba-4fb5-a15a-d26645e949df","Type":"ContainerStarted","Data":"7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc"}
Sep 18 14:18:06.122406 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.121886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7758-453c2e" event={"ID":"42cf7841-19ba-4fb5-a15a-d26645e949df","Type":"ContainerStarted","Data":"f27e8cda154460769380cdd47108957ba452f940a60b9f4b02884945e187217f"}
Sep 18 14:18:06.122406 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.122051    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7758-7be963" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" containerName="test-curl-7758-7be963" containerID="cri-o://defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61" gracePeriod=2
Sep 18 14:18:06.140217 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.140157    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7758-453c2e" podStartSLOduration=2.140121926 podStartE2EDuration="2.140121926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:06.138361106 +0000 UTC m=+1174.294881387" watchObservedRunningTime="2026-09-18 14:18:06.140121926 +0000 UTC m=+1174.296642208"
Sep 18 14:18:06.140738 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:06.140710    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-7be963\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:06.829512 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.829465    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7758-453c2e"]
Sep 18 14:18:06.839040 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.838944    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7758-453c2e"]
Sep 18 14:18:06.963679 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.963633    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7758-b220b7"]
Sep 18 14:18:06.964075 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.964061    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42cf7841-19ba-4fb5-a15a-d26645e949df" containerName="test-curl-7758-453c2e"
Sep 18 14:18:06.964146 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.964076    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="42cf7841-19ba-4fb5-a15a-d26645e949df" containerName="test-curl-7758-453c2e"
Sep 18 14:18:06.964216 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.964203    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="42cf7841-19ba-4fb5-a15a-d26645e949df" containerName="test-curl-7758-453c2e"
Sep 18 14:18:06.967621 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.967594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:06.976490 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:06.976452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7758-b220b7"]
Sep 18 14:18:07.128373 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:07.128280    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7758-453c2e" container="test-curl-7758-453c2e"
Sep 18 14:18:07.139318 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:07.139292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmlv\" (UniqueName: \"kubernetes.io/projected/bcf73138-cb95-447c-a1bf-99ff4f429723-kube-api-access-qxmlv\") pod \"test-curl-7758-b220b7\" (UID: \"bcf73138-cb95-447c-a1bf-99ff4f429723\") " pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:07.240938 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:07.240891    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmlv\" (UniqueName: \"kubernetes.io/projected/bcf73138-cb95-447c-a1bf-99ff4f429723-kube-api-access-qxmlv\") pod \"test-curl-7758-b220b7\" (UID: \"bcf73138-cb95-447c-a1bf-99ff4f429723\") " pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:07.249607 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:07.249573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmlv\" (UniqueName: \"kubernetes.io/projected/bcf73138-cb95-447c-a1bf-99ff4f429723-kube-api-access-qxmlv\") pod \"test-curl-7758-b220b7\" (UID: \"bcf73138-cb95-447c-a1bf-99ff4f429723\") " pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:07.280532 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:07.280486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:07.568876 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:18:07.568840    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbcf73138_cb95_447c_a1bf_99ff4f429723.slice/crio-c60e13fafa012b35dfb0b4d616c1f8bc2a382c5ab669cda625a5030ba0449cdf WatchSource:0}: Error finding container c60e13fafa012b35dfb0b4d616c1f8bc2a382c5ab669cda625a5030ba0449cdf: Status 404 returned error can't find the container with id c60e13fafa012b35dfb0b4d616c1f8bc2a382c5ab669cda625a5030ba0449cdf
Sep 18 14:18:07.569032 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:07.568973    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7758-b220b7"]
Sep 18 14:18:08.137438 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:08.137401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7758-b220b7" event={"ID":"bcf73138-cb95-447c-a1bf-99ff4f429723","Type":"ContainerStarted","Data":"8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513"}
Sep 18 14:18:08.137833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:08.137453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7758-b220b7" event={"ID":"bcf73138-cb95-447c-a1bf-99ff4f429723","Type":"ContainerStarted","Data":"c60e13fafa012b35dfb0b4d616c1f8bc2a382c5ab669cda625a5030ba0449cdf"}
Sep 18 14:18:08.137833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:08.137458    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7758-453c2e" podUID="42cf7841-19ba-4fb5-a15a-d26645e949df" containerName="test-curl-7758-453c2e" containerID="cri-o://7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc" gracePeriod=2
Sep 18 14:18:08.181033 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:08.180958    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7758-b220b7" podStartSLOduration=2.180933868 podStartE2EDuration="2.180933868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:08.180338897 +0000 UTC m=+1176.336859178" watchObservedRunningTime="2026-09-18 14:18:08.180933868 +0000 UTC m=+1176.337454153"
Sep 18 14:18:08.184478 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:08.184432    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-453c2e\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="42cf7841-19ba-4fb5-a15a-d26645e949df" pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:08.187894 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:08.187856    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-453c2e\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="42cf7841-19ba-4fb5-a15a-d26645e949df" pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:08.349121 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:08.349090    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:08.351898 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:08.351870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r6q4\" (UniqueName: \"kubernetes.io/projected/eb0508fc-5d2c-4162-929e-d6b0e88b1866-kube-api-access-9r6q4\") pod \"eb0508fc-5d2c-4162-929e-d6b0e88b1866\" (UID: \"eb0508fc-5d2c-4162-929e-d6b0e88b1866\") "
Sep 18 14:18:08.354157 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:08.354096    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0508fc-5d2c-4162-929e-d6b0e88b1866-kube-api-access-9r6q4" (OuterVolumeSpecName: "kube-api-access-9r6q4") pod "eb0508fc-5d2c-4162-929e-d6b0e88b1866" (UID: "eb0508fc-5d2c-4162-929e-d6b0e88b1866"). InnerVolumeSpecName "kube-api-access-9r6q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:18:08.453387 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:08.453347    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r6q4\" (UniqueName: \"kubernetes.io/projected/eb0508fc-5d2c-4162-929e-d6b0e88b1866-kube-api-access-9r6q4\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:09.153372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.153261    2563 generic.go:356] "Generic (PLEG): container finished" podID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" containerID="defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61" exitCode=137
Sep 18 14:18:09.153372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.153315    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:09.153903 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.153358    2563 scope.go:117] "RemoveContainer" containerID="defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61"
Sep 18 14:18:09.164003 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.163983    2563 scope.go:117] "RemoveContainer" containerID="defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61"
Sep 18 14:18:09.164409 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:09.164383    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61\": container with ID starting with defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61 not found: ID does not exist" containerID="defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61"
Sep 18 14:18:09.164520 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:09.164400    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-7be963\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:09.164520 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.164417    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61"} err="failed to get container status \"defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61\": rpc error: code = NotFound desc = could not find container \"defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61\": container with ID starting with defe474076289b063cb1da8b766cea3d97819cccec5d48a3ad4cef48c6159b61 not found: ID does not exist"
Sep 18 14:18:09.190397 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.190356    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7758-b220b7"]
Sep 18 14:18:09.200810 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.200772    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7758-b220b7"]
Sep 18 14:18:09.313827 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.313781    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7758-3975bc"]
Sep 18 14:18:09.314258 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.314244    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" containerName="test-curl-7758-b220b7"
Sep 18 14:18:09.314312 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.314260    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" containerName="test-curl-7758-b220b7"
Sep 18 14:18:09.314350 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.314335    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" containerName="test-curl-7758-b220b7"
Sep 18 14:18:09.318938 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.318919    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-3975bc"
Sep 18 14:18:09.325388 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.325355    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7758-3975bc"]
Sep 18 14:18:09.339468 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:09.339427    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-7be963\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:09.360033 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.359993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtgxz\" (UniqueName: \"kubernetes.io/projected/a23c2b10-c877-42f5-8846-42f0cbf5d39e-kube-api-access-vtgxz\") pod \"test-curl-7758-3975bc\" (UID: \"a23c2b10-c877-42f5-8846-42f0cbf5d39e\") " pod="opendatahub/test-curl-7758-3975bc"
Sep 18 14:18:09.460953 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.460911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtgxz\" (UniqueName: \"kubernetes.io/projected/a23c2b10-c877-42f5-8846-42f0cbf5d39e-kube-api-access-vtgxz\") pod \"test-curl-7758-3975bc\" (UID: \"a23c2b10-c877-42f5-8846-42f0cbf5d39e\") " pod="opendatahub/test-curl-7758-3975bc"
Sep 18 14:18:09.477936 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.477907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtgxz\" (UniqueName: \"kubernetes.io/projected/a23c2b10-c877-42f5-8846-42f0cbf5d39e-kube-api-access-vtgxz\") pod \"test-curl-7758-3975bc\" (UID: \"a23c2b10-c877-42f5-8846-42f0cbf5d39e\") " pod="opendatahub/test-curl-7758-3975bc"
Sep 18 14:18:09.628096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.628064    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:18:09.630833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.630814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-3975bc"
Sep 18 14:18:09.659923 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:09.659862    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-7be963\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:09.663328 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.663276    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-data\") pod \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") "
Sep 18 14:18:09.663548 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.663358    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-certs\") pod \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") "
Sep 18 14:18:09.663548 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.663404    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-credential-socket\") pod \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") "
Sep 18 14:18:09.663548 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.663443    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istiod-ca-cert\") pod \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") "
Sep 18 14:18:09.663548 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.663497    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-podinfo\") pod \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") "
Sep 18 14:18:09.663792 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.663586    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-token\") pod \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") "
Sep 18 14:18:09.663792 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.663629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-envoy\") pod \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") "
Sep 18 14:18:09.663792 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.663655    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" (UID: "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:18:09.663792 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.663684    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-socket\") pod \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") "
Sep 18 14:18:09.663792 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.663714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5b84\" (UniqueName: \"kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-kube-api-access-k5b84\") pod \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\" (UID: \"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb\") "
Sep 18 14:18:09.664578 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.664049    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" (UID: "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:18:09.664578 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.664197    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-credential-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:09.664578 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.664222    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-data\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:09.664578 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.664221    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" (UID: "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:18:09.664578 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.664246    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" (UID: "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:18:09.665041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.664921    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" (UID: "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:18:09.667492 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.667441    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" (UID: "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:18:09.667661 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.667514    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-kube-api-access-k5b84" (OuterVolumeSpecName: "kube-api-access-k5b84") pod "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" (UID: "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb"). InnerVolumeSpecName "kube-api-access-k5b84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:18:09.667661 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.667644    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" (UID: "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:18:09.667810 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.667723    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" (UID: "82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:18:09.765496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.765388    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:09.765496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.765424    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5b84\" (UniqueName: \"kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-kube-api-access-k5b84\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:09.765496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.765434    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-workload-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:09.765496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.765443    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istiod-ca-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:09.765496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.765454    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-podinfo\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:09.765496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.765465    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-token\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:09.765496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.765473    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb-istio-envoy\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:09.920629 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:09.920596    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7758-3975bc"]
Sep 18 14:18:09.922958 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:18:09.922923    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda23c2b10_c877_42f5_8846_42f0cbf5d39e.slice/crio-1fb238802fcdef3a24fc207e8b07052abeef6a69cc85b238781f66874e8c1d2b WatchSource:0}: Error finding container 1fb238802fcdef3a24fc207e8b07052abeef6a69cc85b238781f66874e8c1d2b: Status 404 returned error can't find the container with id 1fb238802fcdef3a24fc207e8b07052abeef6a69cc85b238781f66874e8c1d2b
Sep 18 14:18:10.122254 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.122121    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:10.122254 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.122209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:10.135647 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.135605    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:10.158041 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:10.157997    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-7be963\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:10.163703 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.163658    2563 generic.go:356] "Generic (PLEG): container finished" podID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerID="ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c" exitCode=0
Sep 18 14:18:10.163927 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.163735    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"
Sep 18 14:18:10.163927 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.163750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" event={"ID":"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb","Type":"ContainerDied","Data":"ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c"}
Sep 18 14:18:10.163927 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.163819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh" event={"ID":"82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb","Type":"ContainerDied","Data":"20b92ee37730f4b46e1ef4a75ea49106b015505ebabf0e896ded7abecbf23e8e"}
Sep 18 14:18:10.163927 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.163854    2563 scope.go:117] "RemoveContainer" containerID="ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c"
Sep 18 14:18:10.165938 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.165903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7758-3975bc" event={"ID":"a23c2b10-c877-42f5-8846-42f0cbf5d39e","Type":"ContainerStarted","Data":"d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe"}
Sep 18 14:18:10.166092 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.165949    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7758-3975bc" event={"ID":"a23c2b10-c877-42f5-8846-42f0cbf5d39e","Type":"ContainerStarted","Data":"1fb238802fcdef3a24fc207e8b07052abeef6a69cc85b238781f66874e8c1d2b"}
Sep 18 14:18:10.166223 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.166192    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7758-b220b7" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" containerName="test-curl-7758-b220b7" containerID="cri-o://8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513" gracePeriod=2
Sep 18 14:18:10.173343 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:10.173309    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-7be963\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:10.175389 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:10.175357    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-b220b7\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:10.179436 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.179413    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:18:10.191758 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:10.191714    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-7be963\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:10.192699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.192651    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7758-3975bc" podStartSLOduration=1.192633546 podStartE2EDuration="1.192633546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:10.18971722 +0000 UTC m=+1178.346237502" watchObservedRunningTime="2026-09-18 14:18:10.192633546 +0000 UTC m=+1178.349153827"
Sep 18 14:18:10.194220 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:10.194180    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-7be963\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:10.202088 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.202060    2563 scope.go:117] "RemoveContainer" containerID="0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d"
Sep 18 14:18:10.213739 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.213506    2563 scope.go:117] "RemoveContainer" containerID="ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c"
Sep 18 14:18:10.214063 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:10.214033    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c\": container with ID starting with ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c not found: ID does not exist" containerID="ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c"
Sep 18 14:18:10.214170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.214080    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c"} err="failed to get container status \"ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c\": rpc error: code = NotFound desc = could not find container \"ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c\": container with ID starting with ef26dd990be32f7933134829d4f8ffdf334d1b428c93cd2de61a639a094f895c not found: ID does not exist"
Sep 18 14:18:10.214170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.214109    2563 scope.go:117] "RemoveContainer" containerID="0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d"
Sep 18 14:18:10.214514 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:10.214485    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d\": container with ID starting with 0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d not found: ID does not exist" containerID="0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d"
Sep 18 14:18:10.214623 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.214517    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d"} err="failed to get container status \"0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d\": rpc error: code = NotFound desc = could not find container \"0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d\": container with ID starting with 0ff31454de5b587c669e870d9452f49f7ee3331470ab63bf24574ea2e82eda2d not found: ID does not exist"
Sep 18 14:18:10.216153 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:10.216100    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-b220b7\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:10.218218 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:10.218192    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-b220b7\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:10.241442 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.241399    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"]
Sep 18 14:18:10.246520 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:10.246477    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-7be963\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" pod="opendatahub/test-curl-7758-7be963"
Sep 18 14:18:10.248539 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.248506    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ee8a08-openshift-default-76fb6bd766-t9rwh"]
Sep 18 14:18:10.292929 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.292883    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" path="/var/lib/kubelet/pods/82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb/volumes"
Sep 18 14:18:10.293326 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.293310    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb0508fc-5d2c-4162-929e-d6b0e88b1866" path="/var/lib/kubelet/pods/eb0508fc-5d2c-4162-929e-d6b0e88b1866/volumes"
Sep 18 14:18:10.313319 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.313290    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:10.371688 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.371653    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jp9s\" (UniqueName: \"kubernetes.io/projected/42cf7841-19ba-4fb5-a15a-d26645e949df-kube-api-access-8jp9s\") pod \"42cf7841-19ba-4fb5-a15a-d26645e949df\" (UID: \"42cf7841-19ba-4fb5-a15a-d26645e949df\") "
Sep 18 14:18:10.373945 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.373877    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cf7841-19ba-4fb5-a15a-d26645e949df-kube-api-access-8jp9s" (OuterVolumeSpecName: "kube-api-access-8jp9s") pod "42cf7841-19ba-4fb5-a15a-d26645e949df" (UID: "42cf7841-19ba-4fb5-a15a-d26645e949df"). InnerVolumeSpecName "kube-api-access-8jp9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:18:10.472719 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.472680    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jp9s\" (UniqueName: \"kubernetes.io/projected/42cf7841-19ba-4fb5-a15a-d26645e949df-kube-api-access-8jp9s\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:10.915240 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.915203    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7758-3975bc"]
Sep 18 14:18:10.923754 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:10.923711    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7758-3975bc"]
Sep 18 14:18:11.173872 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.173831    2563 generic.go:356] "Generic (PLEG): container finished" podID="42cf7841-19ba-4fb5-a15a-d26645e949df" containerID="7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc" exitCode=137
Sep 18 14:18:11.174388 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.173883    2563 scope.go:117] "RemoveContainer" containerID="7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc"
Sep 18 14:18:11.174388 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.173884    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:11.175902 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:11.175875    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7758-3975bc" container="test-curl-7758-3975bc"
Sep 18 14:18:11.186550 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.186522    2563 scope.go:117] "RemoveContainer" containerID="7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc"
Sep 18 14:18:11.186653 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:11.186617    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-453c2e\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="42cf7841-19ba-4fb5-a15a-d26645e949df" pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:11.187037 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:11.187011    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc\": container with ID starting with 7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc not found: ID does not exist" containerID="7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc"
Sep 18 14:18:11.187080 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.187051    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc"} err="failed to get container status \"7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc\": rpc error: code = NotFound desc = could not find container \"7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc\": container with ID starting with 7ad8f530bf2e80ce80313394fa0e6c7415457e3812e45a3b017daf2015e202bc not found: ID does not exist"
Sep 18 14:18:11.870167 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.870095    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"]
Sep 18 14:18:11.870799 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.870775    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23c2b10-c877-42f5-8846-42f0cbf5d39e" containerName="test-curl-7758-3975bc"
Sep 18 14:18:11.870869 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.870805    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23c2b10-c877-42f5-8846-42f0cbf5d39e" containerName="test-curl-7758-3975bc"
Sep 18 14:18:11.870869 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.870824    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy"
Sep 18 14:18:11.870869 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.870833    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy"
Sep 18 14:18:11.870869 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.870847    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy"
Sep 18 14:18:11.870869 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.870856    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy"
Sep 18 14:18:11.871041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.870970    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23c2b10-c877-42f5-8846-42f0cbf5d39e" containerName="test-curl-7758-3975bc"
Sep 18 14:18:11.871041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.870987    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy"
Sep 18 14:18:11.871041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.870996    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ba982a-4a96-4d3c-b8b4-d832bfc2e3fb" containerName="istio-proxy"
Sep 18 14:18:11.875659 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.875634    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.878325 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:11.878288    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-453c2e\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="42cf7841-19ba-4fb5-a15a-d26645e949df" pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:11.878528 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.878509    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 14:18:11.878722 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.878703    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-988c9d38-openshift-default-dockercfg-tbm6q\""
Sep 18 14:18:11.886107 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.886081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-podinfo\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.886354 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.886160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-certs\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.886354 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.886199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-credential-socket\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.886354 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.886245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-data\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.886354 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.886282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-socket\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.886354 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.886332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-envoy\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.886620 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.886369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4f751a-c79c-4c15-b24c-d508367dd676-istiod-ca-cert\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.886620 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.886403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wxlw\" (UniqueName: \"kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-kube-api-access-8wxlw\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.886620 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.886443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-token\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.887085 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.887063    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"]
Sep 18 14:18:11.987739 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.987697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-certs\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.987739 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.987746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-credential-socket\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.987784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-data\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.987808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-socket\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.987848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-envoy\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.987879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4f751a-c79c-4c15-b24c-d508367dd676-istiod-ca-cert\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.987900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxlw\" (UniqueName: \"kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-kube-api-access-8wxlw\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.987929    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-token\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.988244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-certs\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.988244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-credential-socket\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.988442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-socket\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988933 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.988441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-podinfo\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988933 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.988631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-data\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.988933 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.988837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4f751a-c79c-4c15-b24c-d508367dd676-istiod-ca-cert\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.990562 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.990536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-envoy\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.991352 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.991329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-podinfo\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.998166 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.998103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wxlw\" (UniqueName: \"kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-kube-api-access-8wxlw\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:11.998717 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:11.998691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-token\") pod \"e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") " pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:12.184085 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:12.184047    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7758-3975bc" podUID="a23c2b10-c877-42f5-8846-42f0cbf5d39e" containerName="test-curl-7758-3975bc" containerID="cri-o://d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe" gracePeriod=2
Sep 18 14:18:12.186897 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:12.186862    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-453c2e\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="42cf7841-19ba-4fb5-a15a-d26645e949df" pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:12.188932 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:12.188909    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-3975bc\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="a23c2b10-c877-42f5-8846-42f0cbf5d39e" pod="opendatahub/test-curl-7758-3975bc"
Sep 18 14:18:12.190100 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:12.190067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:12.295920 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:12.295874    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-3975bc\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="a23c2b10-c877-42f5-8846-42f0cbf5d39e" pod="opendatahub/test-curl-7758-3975bc"
Sep 18 14:18:12.298507 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:12.298463    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-453c2e\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="42cf7841-19ba-4fb5-a15a-d26645e949df" pod="opendatahub/test-curl-7758-453c2e"
Sep 18 14:18:12.299964 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:12.299939    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42cf7841-19ba-4fb5-a15a-d26645e949df" path="/var/lib/kubelet/pods/42cf7841-19ba-4fb5-a15a-d26645e949df/volumes"
Sep 18 14:18:12.379873 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:12.379841    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:12.382160 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:12.382107    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-b220b7\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:12.392559 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:12.392528    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxmlv\" (UniqueName: \"kubernetes.io/projected/bcf73138-cb95-447c-a1bf-99ff4f429723-kube-api-access-qxmlv\") pod \"bcf73138-cb95-447c-a1bf-99ff4f429723\" (UID: \"bcf73138-cb95-447c-a1bf-99ff4f429723\") "
Sep 18 14:18:12.394804 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:12.394772    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf73138-cb95-447c-a1bf-99ff4f429723-kube-api-access-qxmlv" (OuterVolumeSpecName: "kube-api-access-qxmlv") pod "bcf73138-cb95-447c-a1bf-99ff4f429723" (UID: "bcf73138-cb95-447c-a1bf-99ff4f429723"). InnerVolumeSpecName "kube-api-access-qxmlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:18:12.491276 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:12.491240    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"]
Sep 18 14:18:12.492437 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:18:12.492408    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4f751a_c79c_4c15_b24c_d508367dd676.slice/crio-faae42fa9ba8e2e0bd2b5deec8f7e0540cb487dcb76eea5257e4307e51d5ed54 WatchSource:0}: Error finding container faae42fa9ba8e2e0bd2b5deec8f7e0540cb487dcb76eea5257e4307e51d5ed54: Status 404 returned error can't find the container with id faae42fa9ba8e2e0bd2b5deec8f7e0540cb487dcb76eea5257e4307e51d5ed54
Sep 18 14:18:12.493628 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:12.493607    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxmlv\" (UniqueName: \"kubernetes.io/projected/bcf73138-cb95-447c-a1bf-99ff4f429723-kube-api-access-qxmlv\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:12.494865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:12.494826    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:18:12.494977 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:12.494919    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:18:12.494977 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:12.494970    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:18:13.108755 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.108715    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"]
Sep 18 14:18:13.112520 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.112492    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.115552 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:13.115370    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-b220b7\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:13.115725 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.115667    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-ae54af-serving-cert\""
Sep 18 14:18:13.127001 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.126970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"]
Sep 18 14:18:13.193965 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.193925    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" event={"ID":"dd4f751a-c79c-4c15-b24c-d508367dd676","Type":"ContainerStarted","Data":"f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7"}
Sep 18 14:18:13.194497 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.193992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" event={"ID":"dd4f751a-c79c-4c15-b24c-d508367dd676","Type":"ContainerStarted","Data":"faae42fa9ba8e2e0bd2b5deec8f7e0540cb487dcb76eea5257e4307e51d5ed54"}
Sep 18 14:18:13.195402 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.195377    2563 generic.go:356] "Generic (PLEG): container finished" podID="bcf73138-cb95-447c-a1bf-99ff4f429723" containerID="8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513" exitCode=137
Sep 18 14:18:13.195508 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.195423    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:13.195508 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.195471    2563 scope.go:117] "RemoveContainer" containerID="8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513"
Sep 18 14:18:13.196089 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:13.196056    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-b220b7\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:13.200835 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.200814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.200952 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.200872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.200952 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.200898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b414230-c641-4521-b5e1-03f285fedbef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.201043 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.200970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnlk\" (UniqueName: \"kubernetes.io/projected/7b414230-c641-4521-b5e1-03f285fedbef-kube-api-access-9qnlk\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.205595 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.205571    2563 scope.go:117] "RemoveContainer" containerID="8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513"
Sep 18 14:18:13.205953 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:13.205932    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513\": container with ID starting with 8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513 not found: ID does not exist" containerID="8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513"
Sep 18 14:18:13.206013 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.205963    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513"} err="failed to get container status \"8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513\": rpc error: code = NotFound desc = could not find container \"8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513\": container with ID starting with 8680a80f744a9ce547e3aa1a79a1c28f4ff86d4fa5902c9711e69d5c1092f513 not found: ID does not exist"
Sep 18 14:18:13.221246 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.221186    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" podStartSLOduration=2.221166214 podStartE2EDuration="2.221166214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:13.220187548 +0000 UTC m=+1181.376707852" watchObservedRunningTime="2026-09-18 14:18:13.221166214 +0000 UTC m=+1181.377686495"
Sep 18 14:18:13.222462 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:13.222424    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-b220b7\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:13.301982 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.301872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.301982 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.301950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.301982 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.301971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b414230-c641-4521-b5e1-03f285fedbef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.302376 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:13.302017    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-serving-cert: secret "maas-api-e2e-shared-a-w3-ae54af-serving-cert" not found
Sep 18 14:18:13.302376 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.302037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnlk\" (UniqueName: \"kubernetes.io/projected/7b414230-c641-4521-b5e1-03f285fedbef-kube-api-access-9qnlk\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.302376 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:13.302099    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-maas-api-tls podName:7b414230-c641-4521-b5e1-03f285fedbef nodeName:}" failed. No retries permitted until 2026-09-18 14:18:13.802074175 +0000 UTC m=+1181.958594452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-maas-api-tls") pod "maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk" (UID: "7b414230-c641-4521-b5e1-03f285fedbef") : secret "maas-api-e2e-shared-a-w3-ae54af-serving-cert" not found
Sep 18 14:18:13.302812 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.302791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b414230-c641-4521-b5e1-03f285fedbef-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.304665 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.304637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.314022 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.313806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnlk\" (UniqueName: \"kubernetes.io/projected/7b414230-c641-4521-b5e1-03f285fedbef-kube-api-access-9qnlk\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.806732 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.806691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:13.809666 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:13.809628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:14.024797 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:14.024716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:14.191032 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:14.190787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:14.197017 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:14.196985    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:14.203731 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:14.203703    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:14.204714 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:14.204691    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:14.224860 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:14.224821    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-b220b7\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:14.226896 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:14.226862    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-b220b7\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" pod="opendatahub/test-curl-7758-b220b7"
Sep 18 14:18:14.290776 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:14.290739    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcf73138-cb95-447c-a1bf-99ff4f429723" path="/var/lib/kubelet/pods/bcf73138-cb95-447c-a1bf-99ff4f429723/volumes"
Sep 18 14:18:14.308422 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:14.308395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"]
Sep 18 14:18:14.312219 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:18:14.312184    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b414230_c641_4521_b5e1_03f285fedbef.slice/crio-a1277c40b564362c18ece59245b65c28fae2f8edd7dbc618d9891a84df67a0cf WatchSource:0}: Error finding container a1277c40b564362c18ece59245b65c28fae2f8edd7dbc618d9891a84df67a0cf: Status 404 returned error can't find the container with id a1277c40b564362c18ece59245b65c28fae2f8edd7dbc618d9891a84df67a0cf
Sep 18 14:18:14.392324 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:14.392295    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-3975bc"
Sep 18 14:18:14.415293 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:14.415259    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtgxz\" (UniqueName: \"kubernetes.io/projected/a23c2b10-c877-42f5-8846-42f0cbf5d39e-kube-api-access-vtgxz\") pod \"a23c2b10-c877-42f5-8846-42f0cbf5d39e\" (UID: \"a23c2b10-c877-42f5-8846-42f0cbf5d39e\") "
Sep 18 14:18:14.417709 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:14.417681    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23c2b10-c877-42f5-8846-42f0cbf5d39e-kube-api-access-vtgxz" (OuterVolumeSpecName: "kube-api-access-vtgxz") pod "a23c2b10-c877-42f5-8846-42f0cbf5d39e" (UID: "a23c2b10-c877-42f5-8846-42f0cbf5d39e"). InnerVolumeSpecName "kube-api-access-vtgxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:18:14.516661 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:14.516616    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtgxz\" (UniqueName: \"kubernetes.io/projected/a23c2b10-c877-42f5-8846-42f0cbf5d39e-kube-api-access-vtgxz\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:18:15.212561 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:15.212506    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk" event={"ID":"7b414230-c641-4521-b5e1-03f285fedbef","Type":"ContainerStarted","Data":"61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729"}
Sep 18 14:18:15.213041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:15.212729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk" event={"ID":"7b414230-c641-4521-b5e1-03f285fedbef","Type":"ContainerStarted","Data":"a1277c40b564362c18ece59245b65c28fae2f8edd7dbc618d9891a84df67a0cf"}
Sep 18 14:18:15.213041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:15.212790    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:15.214264 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:15.214232    2563 generic.go:356] "Generic (PLEG): container finished" podID="a23c2b10-c877-42f5-8846-42f0cbf5d39e" containerID="d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe" exitCode=137
Sep 18 14:18:15.214466 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:15.214277    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7758-3975bc"
Sep 18 14:18:15.214466 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:15.214328    2563 scope.go:117] "RemoveContainer" containerID="d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe"
Sep 18 14:18:15.225701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:15.225545    2563 scope.go:117] "RemoveContainer" containerID="d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe"
Sep 18 14:18:15.225874 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:15.225855    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe\": container with ID starting with d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe not found: ID does not exist" containerID="d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe"
Sep 18 14:18:15.225920 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:15.225885    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe"} err="failed to get container status \"d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe\": rpc error: code = NotFound desc = could not find container \"d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe\": container with ID starting with d7c48094db81a588b6a60fd7e327f83e7a1c0e5198b7d4fab5a71bbe73c4d9fe not found: ID does not exist"
Sep 18 14:18:15.256263 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:15.256219    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7758-3975bc\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="a23c2b10-c877-42f5-8846-42f0cbf5d39e" pod="opendatahub/test-curl-7758-3975bc"
Sep 18 14:18:15.256539 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:15.256496    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk" podStartSLOduration=1.774652854 podStartE2EDuration="2.256479792s" totalImagesPullingTime="481.826938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:15.247915081 +0000 UTC m=+1183.404435363" watchObservedRunningTime="2026-09-18 14:18:15.256479792 +0000 UTC m=+1183.413000073"
Sep 18 14:18:16.288918 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:16.288880    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23c2b10-c877-42f5-8846-42f0cbf5d39e" path="/var/lib/kubelet/pods/a23c2b10-c877-42f5-8846-42f0cbf5d39e/volumes"
Sep 18 14:18:21.230142 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:21.230098    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:18:28.528677 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:28.528627    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:18:32.699035 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:32.698937    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:18:32.728567 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:32.728527    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:18:40.391021 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:40.390980    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerID="f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7" exitCode=0
Sep 18 14:18:40.391553 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:40.391053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" event={"ID":"dd4f751a-c79c-4c15-b24c-d508367dd676","Type":"ContainerDied","Data":"f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7"}
Sep 18 14:18:40.391553 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:40.391526    2563 scope.go:117] "RemoveContainer" containerID="f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7"
Sep 18 14:18:40.392853 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:40.392814    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:18:40.392994 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:40.392910    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:18:40.392994 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:40.392961    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:18:41.399919 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:41.399879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" event={"ID":"dd4f751a-c79c-4c15-b24c-d508367dd676","Type":"ContainerStarted","Data":"e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9"}
Sep 18 14:18:42.190945 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:42.190902    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:42.192551 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:42.192516    2563 patch_prober.go:28] interesting pod/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 18 14:18:42.192713 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:42.192570    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 18 14:18:43.191003 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:43.190964    2563 patch_prober.go:28] interesting pod/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 18 14:18:43.191439 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:43.191042    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 18 14:18:44.191467 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.191412    2563 patch_prober.go:28] interesting pod/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 18 14:18:44.191901 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.191492    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 18 14:18:44.203918 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.203866    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:44.736499 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.736448    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"]
Sep 18 14:18:44.739175 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.739120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.741923 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.741895    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-93827d\"/\"distinct-w4-93827d-kserve-self-signed-certs\""
Sep 18 14:18:44.750377 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.750341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7hfl\" (UniqueName: \"kubernetes.io/projected/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-kube-api-access-k7hfl\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.750544 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.750410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-dshm\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.750623 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.750546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tls-certs\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.750679 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.750655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tmp-dir\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.750752 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.750725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-model-cache\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.750815 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.750756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-home\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.751589 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.751560    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"]
Sep 18 14:18:44.851891 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.851853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7hfl\" (UniqueName: \"kubernetes.io/projected/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-kube-api-access-k7hfl\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.852149 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.851913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-dshm\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.852149 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.851958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tls-certs\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.852149 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.852021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tmp-dir\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.852149 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.852056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-model-cache\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.852149 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.852075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-home\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.852555 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.852528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tmp-dir\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.852555 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.852548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-model-cache\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.852686 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.852583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-home\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.854555 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.854534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-dshm\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.854770 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.854747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tls-certs\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:44.860766 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:44.860743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7hfl\" (UniqueName: \"kubernetes.io/projected/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-kube-api-access-k7hfl\") pod \"distinct-w4-93827d-kserve-59d8464bcd-d9zks\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:45.051014 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:45.050896    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:45.190787 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:45.190744    2563 patch_prober.go:28] interesting pod/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 18 14:18:45.190999 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:45.190814    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 18 14:18:45.410196 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:45.410050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"]
Sep 18 14:18:45.431111 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:18:45.430974    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ecbdab2_1c44_4d9d_9cb0_7e40f3f009fb.slice/crio-e25d7fcf33ad6d1a9fe0d48cfcfa367da433af1a4136586fc72f9a0fa95b7b1e WatchSource:0}: Error finding container e25d7fcf33ad6d1a9fe0d48cfcfa367da433af1a4136586fc72f9a0fa95b7b1e: Status 404 returned error can't find the container with id e25d7fcf33ad6d1a9fe0d48cfcfa367da433af1a4136586fc72f9a0fa95b7b1e
Sep 18 14:18:45.442576 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:45.442542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks" event={"ID":"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb","Type":"ContainerStarted","Data":"e25d7fcf33ad6d1a9fe0d48cfcfa367da433af1a4136586fc72f9a0fa95b7b1e"}
Sep 18 14:18:46.190582 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:46.190535    2563 patch_prober.go:28] interesting pod/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 18 14:18:46.190802 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:46.190601    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 18 14:18:46.451388 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:46.451284    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks" event={"ID":"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb","Type":"ContainerStarted","Data":"e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888"}
Sep 18 14:18:46.472712 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:46.472652    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks" podStartSLOduration=2.472632413 podStartE2EDuration="2.472632413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:46.471277544 +0000 UTC m=+1214.627797828" watchObservedRunningTime="2026-09-18 14:18:46.472632413 +0000 UTC m=+1214.629152693"
Sep 18 14:18:47.195919 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.195870    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:47.196997 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.196971    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:18:47.799459 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.799422    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"]
Sep 18 14:18:47.802256 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.802226    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:47.805535 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.805460    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-980661-serving-cert\""
Sep 18 14:18:47.818938 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.818907    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"]
Sep 18 14:18:47.884714 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.884670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a445d889-bf7f-4451-89d7-f0e0bc4345e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:47.884919 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.884719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm94d\" (UniqueName: \"kubernetes.io/projected/a445d889-bf7f-4451-89d7-f0e0bc4345e3-kube-api-access-gm94d\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:47.884919 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.884776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:47.884919 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.884859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:47.986235 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.986196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a445d889-bf7f-4451-89d7-f0e0bc4345e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:47.986235 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.986238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm94d\" (UniqueName: \"kubernetes.io/projected/a445d889-bf7f-4451-89d7-f0e0bc4345e3-kube-api-access-gm94d\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:47.986542 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.986347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:47.986542 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.986419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:47.986542 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:47.986471    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-serving-cert: secret "maas-api-e2e-shared-b-w3-980661-serving-cert" not found
Sep 18 14:18:47.986709 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:18:47.986569    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-maas-api-tls podName:a445d889-bf7f-4451-89d7-f0e0bc4345e3 nodeName:}" failed. No retries permitted until 2026-09-18 14:18:48.486527373 +0000 UTC m=+1216.643047637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-maas-api-tls") pod "maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb" (UID: "a445d889-bf7f-4451-89d7-f0e0bc4345e3") : secret "maas-api-e2e-shared-b-w3-980661-serving-cert" not found
Sep 18 14:18:47.987036 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.987010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a445d889-bf7f-4451-89d7-f0e0bc4345e3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:47.989188 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:47.989124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:48.004727 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:48.004687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm94d\" (UniqueName: \"kubernetes.io/projected/a445d889-bf7f-4451-89d7-f0e0bc4345e3-kube-api-access-gm94d\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:48.492561 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:48.492516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:48.495224 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:48.495185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:48.716937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:48.716895    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:49.000938 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:49.000906    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"]
Sep 18 14:18:49.003560 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:18:49.003531    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda445d889_bf7f_4451_89d7_f0e0bc4345e3.slice/crio-ce940721d21907a65ee2b8f422c81e2dc260350dc9681e15f63f91d3f38f9314 WatchSource:0}: Error finding container ce940721d21907a65ee2b8f422c81e2dc260350dc9681e15f63f91d3f38f9314: Status 404 returned error can't find the container with id ce940721d21907a65ee2b8f422c81e2dc260350dc9681e15f63f91d3f38f9314
Sep 18 14:18:49.472214 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:49.472174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb" event={"ID":"a445d889-bf7f-4451-89d7-f0e0bc4345e3","Type":"ContainerStarted","Data":"ce940721d21907a65ee2b8f422c81e2dc260350dc9681e15f63f91d3f38f9314"}
Sep 18 14:18:50.480595 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:50.480553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb" event={"ID":"a445d889-bf7f-4451-89d7-f0e0bc4345e3","Type":"ContainerStarted","Data":"0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468"}
Sep 18 14:18:50.481069 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:50.480647    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:50.512848 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:50.512787    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb" podStartSLOduration=2.987705208 podStartE2EDuration="3.51276639s" totalImagesPullingTime="525.061182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:18:50.51159919 +0000 UTC m=+1218.668119473" watchObservedRunningTime="2026-09-18 14:18:50.51276639 +0000 UTC m=+1218.669286671"
Sep 18 14:18:55.051396 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:55.051346    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:55.051916 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:55.051427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:55.064379 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:55.064345    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:55.528190 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:55.528155    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:18:56.496738 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:56.496701    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:18:59.398004 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:18:59.397959    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:19:18.804993 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.804946    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"]
Sep 18 14:19:18.809152 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.809112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.813601 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.813562    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-988c9d38\"/\"e2e-lifecycle-model-988c9d38-kserve-self-signed-certs\""
Sep 18 14:19:18.813601 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.813563    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-988c9d38\"/\"kube-root-ca.crt\""
Sep 18 14:19:18.813865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.813563    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-988c9d38\"/\"default-dockercfg-w82n4\""
Sep 18 14:19:18.813865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.813576    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-988c9d38\"/\"openshift-service-ca.crt\""
Sep 18 14:19:18.817092 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.817058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-dshm\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.817679 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.817569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-home\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.817754 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.817676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-model-cache\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.817820 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.817793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-tmp-dir\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.817881 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.817837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fc84eb-4f27-4f00-8363-da99ad504306-tls-certs\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.817972 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.817933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2k29\" (UniqueName: \"kubernetes.io/projected/69fc84eb-4f27-4f00-8363-da99ad504306-kube-api-access-m2k29\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.820276 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.820247    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"]
Sep 18 14:19:18.918979 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.918933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-model-cache\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.919313 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.919011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-tmp-dir\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.919313 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.919046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fc84eb-4f27-4f00-8363-da99ad504306-tls-certs\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.919313 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.919111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2k29\" (UniqueName: \"kubernetes.io/projected/69fc84eb-4f27-4f00-8363-da99ad504306-kube-api-access-m2k29\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.919313 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.919171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-dshm\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.919313 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.919230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-home\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.919640 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.919509    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-tmp-dir\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.919738 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.919717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-home\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.919799 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.919749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-model-cache\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.922417 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.922387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-dshm\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.922658 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.922639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fc84eb-4f27-4f00-8363-da99ad504306-tls-certs\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:18.929063 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:18.929027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2k29\" (UniqueName: \"kubernetes.io/projected/69fc84eb-4f27-4f00-8363-da99ad504306-kube-api-access-m2k29\") pod \"e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") " pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:19.121744 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:19.121626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:19.410573 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:19.410532    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"]
Sep 18 14:19:19.413164 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:19:19.413106    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69fc84eb_4f27_4f00_8363_da99ad504306.slice/crio-beb7bee62909b1012d7aa94e0a593060be5c5dacf67693046462dae0207e4560 WatchSource:0}: Error finding container beb7bee62909b1012d7aa94e0a593060be5c5dacf67693046462dae0207e4560: Status 404 returned error can't find the container with id beb7bee62909b1012d7aa94e0a593060be5c5dacf67693046462dae0207e4560
Sep 18 14:19:19.678560 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:19.678517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff" event={"ID":"69fc84eb-4f27-4f00-8363-da99ad504306","Type":"ContainerStarted","Data":"c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d"}
Sep 18 14:19:19.678560 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:19.678571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff" event={"ID":"69fc84eb-4f27-4f00-8363-da99ad504306","Type":"ContainerStarted","Data":"beb7bee62909b1012d7aa94e0a593060be5c5dacf67693046462dae0207e4560"}
Sep 18 14:19:19.710193 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:19.703439    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff" podStartSLOduration=1.703412062 podStartE2EDuration="1.703412062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:19:19.699079742 +0000 UTC m=+1247.855600023" watchObservedRunningTime="2026-09-18 14:19:19.703412062 +0000 UTC m=+1247.859932343"
Sep 18 14:19:29.122329 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:29.122285    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:29.122329 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:29.122345    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:29.135501 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:29.135468    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:29.756485 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:29.756455    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:19:32.483977 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:32.483940    2563 scope.go:117] "RemoveContainer" containerID="27989beefcef20a6f9d167a2eeb15164a6840b6e14075be0212141e1fc24c4fc"
Sep 18 14:19:36.335692 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:19:36.335650    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:20:02.543098 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.543048    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"]
Sep 18 14:20:02.546717 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.546694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.549854 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.549833    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ae54af\"/\"default-dockercfg-k6q9w\""
Sep 18 14:20:02.550413 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.550394    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ae54af\"/\"kube-root-ca.crt\""
Sep 18 14:20:02.550460 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.550417    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ae54af\"/\"openshift-service-ca.crt\""
Sep 18 14:20:02.550496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.550401    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ae54af\"/\"auth-test-model-ae54af-kserve-self-signed-certs\""
Sep 18 14:20:02.558243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.558217    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"]
Sep 18 14:20:02.564626 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.564598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-dshm\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.564752 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.564639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tls-certs\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.564752 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.564675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-model-cache\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.564752 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.564704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tmp-dir\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.564878 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.564756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-home\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.564878 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.564791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkmn6\" (UniqueName: \"kubernetes.io/projected/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-kube-api-access-wkmn6\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.666355 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.666320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-dshm\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.666355 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.666354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tls-certs\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.666623 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.666380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-model-cache\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.666623 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.666405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tmp-dir\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.666623 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.666430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-home\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.666623 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.666454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkmn6\" (UniqueName: \"kubernetes.io/projected/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-kube-api-access-wkmn6\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.666846 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.666831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-home\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.666902 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.666873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-model-cache\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.666953 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.666918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tmp-dir\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.668843 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.668822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-dshm\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.669032 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.669016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tls-certs\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.675884 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.675861    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkmn6\" (UniqueName: \"kubernetes.io/projected/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-kube-api-access-wkmn6\") pod \"auth-test-model-ae54af-kserve-859dff9959-bkf6c\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:02.857363 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:02.857275    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:03.141591 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:03.141560    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"]
Sep 18 14:20:03.143006 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:20:03.142976    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b4eaf94_8355_4df0_9f78_cbcc43ec6142.slice/crio-012570afd0f8d64b9ca37364b79814cd054f948e62f748573abffc98d4cf72a7 WatchSource:0}: Error finding container 012570afd0f8d64b9ca37364b79814cd054f948e62f748573abffc98d4cf72a7: Status 404 returned error can't find the container with id 012570afd0f8d64b9ca37364b79814cd054f948e62f748573abffc98d4cf72a7
Sep 18 14:20:03.972670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:03.972631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c" event={"ID":"2b4eaf94-8355-4df0-9f78-cbcc43ec6142","Type":"ContainerStarted","Data":"64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50"}
Sep 18 14:20:03.972670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:03.972676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c" event={"ID":"2b4eaf94-8355-4df0-9f78-cbcc43ec6142","Type":"ContainerStarted","Data":"012570afd0f8d64b9ca37364b79814cd054f948e62f748573abffc98d4cf72a7"}
Sep 18 14:20:03.994997 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:03.994945    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c" podStartSLOduration=1.994928976 podStartE2EDuration="1.994928976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:20:03.992330174 +0000 UTC m=+1292.148850455" watchObservedRunningTime="2026-09-18 14:20:03.994928976 +0000 UTC m=+1292.151449257"
Sep 18 14:20:09.524816 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:09.524775    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:20:12.858289 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:12.858251    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:12.858766 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:12.858340    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:12.870467 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:12.870440    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:13.041845 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:13.041815    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:20:32.508192 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:32.508159    2563 scope.go:117] "RemoveContainer" containerID="862f06b558ba1ba2f99fb57f5e2b1fc71ca344848cde38a34f0dac3ce0edb7a6"
Sep 18 14:20:37.594262 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.594219    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"]
Sep 18 14:20:37.599404 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.599383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.602777 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.602749    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-93827d\"/\"distinct-2-w4-93827d-kserve-self-signed-certs\""
Sep 18 14:20:37.611778 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.611741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-home\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.611926 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.611799    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-model-cache\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.611972 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.611913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1311a7e-85fa-48f1-b159-12119da8827d-tls-certs\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.611972 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.611958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-dshm\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.612041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.612004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-tmp-dir\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.612041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.612023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ndg\" (UniqueName: \"kubernetes.io/projected/d1311a7e-85fa-48f1-b159-12119da8827d-kube-api-access-h2ndg\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.616907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.616865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"]
Sep 18 14:20:37.713249 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.713205    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-dshm\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.713471 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.713272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-tmp-dir\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.713471 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.713297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ndg\" (UniqueName: \"kubernetes.io/projected/d1311a7e-85fa-48f1-b159-12119da8827d-kube-api-access-h2ndg\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.713471 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.713362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-home\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.713471 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.713421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-model-cache\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.713910 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.713532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1311a7e-85fa-48f1-b159-12119da8827d-tls-certs\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.713910 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.713879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-tmp-dir\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.714051 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.714007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-home\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.714051 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.714016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-model-cache\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.715972 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.715948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-dshm\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.716401 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.716378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1311a7e-85fa-48f1-b159-12119da8827d-tls-certs\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.733915 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.733874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ndg\" (UniqueName: \"kubernetes.io/projected/d1311a7e-85fa-48f1-b159-12119da8827d-kube-api-access-h2ndg\") pod \"distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") " pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:37.909750 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:37.909638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:38.223647 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:38.223602    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"]
Sep 18 14:20:38.225575 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:20:38.225544    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1311a7e_85fa_48f1_b159_12119da8827d.slice/crio-c827f8286f1eb2d0621162e445a9aa5418a8be15735747b336db6c0c83b13adb WatchSource:0}: Error finding container c827f8286f1eb2d0621162e445a9aa5418a8be15735747b336db6c0c83b13adb: Status 404 returned error can't find the container with id c827f8286f1eb2d0621162e445a9aa5418a8be15735747b336db6c0c83b13adb
Sep 18 14:20:39.198069 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:39.198028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb" event={"ID":"d1311a7e-85fa-48f1-b159-12119da8827d","Type":"ContainerStarted","Data":"61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288"}
Sep 18 14:20:39.198507 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:39.198077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb" event={"ID":"d1311a7e-85fa-48f1-b159-12119da8827d","Type":"ContainerStarted","Data":"c827f8286f1eb2d0621162e445a9aa5418a8be15735747b336db6c0c83b13adb"}
Sep 18 14:20:39.225378 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:39.225317    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb" podStartSLOduration=2.225298968 podStartE2EDuration="2.225298968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:20:39.224745029 +0000 UTC m=+1327.381265322" watchObservedRunningTime="2026-09-18 14:20:39.225298968 +0000 UTC m=+1327.381819309"
Sep 18 14:20:40.947562 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:40.947518    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:20:47.910396 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:47.910351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:47.910853 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:47.910412    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:47.923320 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:47.923279    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:48.269532 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:48.269494    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:20:49.892499 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:49.892461    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6499b5bf89-8gs7t"]
Sep 18 14:20:49.896509 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:49.896487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:49.920979 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:49.920929    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6499b5bf89-8gs7t"]
Sep 18 14:20:49.954983 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:49.954941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-console-config\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:49.955233 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:49.955015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef400e89-77bb-4aed-8c10-090cbd822016-console-oauth-config\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:49.955233 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:49.955065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-oauth-serving-cert\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:49.955233 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:49.955097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef400e89-77bb-4aed-8c10-090cbd822016-console-serving-cert\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:49.955233 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:49.955121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-trusted-ca-bundle\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:49.955393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:49.955235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq99z\" (UniqueName: \"kubernetes.io/projected/ef400e89-77bb-4aed-8c10-090cbd822016-kube-api-access-dq99z\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:49.955393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:49.955267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-service-ca\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.056888 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.056839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-oauth-serving-cert\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.056888 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.056896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef400e89-77bb-4aed-8c10-090cbd822016-console-serving-cert\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.057204 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.056923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-trusted-ca-bundle\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.057204 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.056983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq99z\" (UniqueName: \"kubernetes.io/projected/ef400e89-77bb-4aed-8c10-090cbd822016-kube-api-access-dq99z\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.057204 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.057014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-service-ca\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.057204 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.057069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-console-config\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.057204 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.057110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef400e89-77bb-4aed-8c10-090cbd822016-console-oauth-config\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.057780 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.057751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-oauth-serving-cert\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.057780 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.057771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-console-config\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.057959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.057876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-service-ca\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.057959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.057881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef400e89-77bb-4aed-8c10-090cbd822016-trusted-ca-bundle\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.059552 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.059528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ef400e89-77bb-4aed-8c10-090cbd822016-console-oauth-config\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.059752 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.059735    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ef400e89-77bb-4aed-8c10-090cbd822016-console-serving-cert\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.070369 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.070339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq99z\" (UniqueName: \"kubernetes.io/projected/ef400e89-77bb-4aed-8c10-090cbd822016-kube-api-access-dq99z\") pod \"console-6499b5bf89-8gs7t\" (UID: \"ef400e89-77bb-4aed-8c10-090cbd822016\") " pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.207092 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.207048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:20:50.494408 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:50.494377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6499b5bf89-8gs7t"]
Sep 18 14:20:50.495655 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:20:50.495619    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef400e89_77bb_4aed_8c10_090cbd822016.slice/crio-f16e822bec07addfb9ee91af7de785c249b0034fdc26c8dea549c742fe5abb24 WatchSource:0}: Error finding container f16e822bec07addfb9ee91af7de785c249b0034fdc26c8dea549c742fe5abb24: Status 404 returned error can't find the container with id f16e822bec07addfb9ee91af7de785c249b0034fdc26c8dea549c742fe5abb24
Sep 18 14:20:51.281730 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:51.281688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6499b5bf89-8gs7t" event={"ID":"ef400e89-77bb-4aed-8c10-090cbd822016","Type":"ContainerStarted","Data":"8f0fa71397b242e56e77002e6098c13fa441feb01d2e328d3c06bc54450e6e54"}
Sep 18 14:20:51.281730 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:51.281738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6499b5bf89-8gs7t" event={"ID":"ef400e89-77bb-4aed-8c10-090cbd822016","Type":"ContainerStarted","Data":"f16e822bec07addfb9ee91af7de785c249b0034fdc26c8dea549c742fe5abb24"}
Sep 18 14:20:51.311760 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:20:51.311697    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6499b5bf89-8gs7t" podStartSLOduration=2.311661855 podStartE2EDuration="2.311661855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:20:51.309908914 +0000 UTC m=+1339.466429185" watchObservedRunningTime="2026-09-18 14:20:51.311661855 +0000 UTC m=+1339.468182216"
Sep 18 14:21:00.207442 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:00.207341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:21:00.208444 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:00.207461    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:21:00.212766 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:00.212735    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:21:00.344250 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:00.344214    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6499b5bf89-8gs7t"
Sep 18 14:21:00.404437 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:00.404380    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f7bbbb964-hx9pt"]
Sep 18 14:21:07.623537 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:07.623498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:21:12.008869 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:12.008825    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"]
Sep 18 14:21:25.431724 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.431676    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f7bbbb964-hx9pt" podUID="f4feab4f-fead-4471-97da-6393e85ad441" containerName="console" containerID="cri-o://b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a" gracePeriod=15
Sep 18 14:21:25.712806 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.712776    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:21:25.859058 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-oauth-serving-cert\") pod \"f4feab4f-fead-4471-97da-6393e85ad441\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") "
Sep 18 14:21:25.859289 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859071    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-service-ca\") pod \"f4feab4f-fead-4471-97da-6393e85ad441\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") "
Sep 18 14:21:25.859289 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jvwn\" (UniqueName: \"kubernetes.io/projected/f4feab4f-fead-4471-97da-6393e85ad441-kube-api-access-9jvwn\") pod \"f4feab4f-fead-4471-97da-6393e85ad441\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") "
Sep 18 14:21:25.859289 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859265    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-trusted-ca-bundle\") pod \"f4feab4f-fead-4471-97da-6393e85ad441\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") "
Sep 18 14:21:25.859468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859321    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-oauth-config\") pod \"f4feab4f-fead-4471-97da-6393e85ad441\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") "
Sep 18 14:21:25.859468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859364    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-serving-cert\") pod \"f4feab4f-fead-4471-97da-6393e85ad441\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") "
Sep 18 14:21:25.859468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-console-config\") pod \"f4feab4f-fead-4471-97da-6393e85ad441\" (UID: \"f4feab4f-fead-4471-97da-6393e85ad441\") "
Sep 18 14:21:25.859635 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859518    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f4feab4f-fead-4471-97da-6393e85ad441" (UID: "f4feab4f-fead-4471-97da-6393e85ad441"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:21:25.859635 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859605    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-service-ca" (OuterVolumeSpecName: "service-ca") pod "f4feab4f-fead-4471-97da-6393e85ad441" (UID: "f4feab4f-fead-4471-97da-6393e85ad441"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:21:25.859801 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859778    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f4feab4f-fead-4471-97da-6393e85ad441" (UID: "f4feab4f-fead-4471-97da-6393e85ad441"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:21:25.859903 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859884    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-console-config" (OuterVolumeSpecName: "console-config") pod "f4feab4f-fead-4471-97da-6393e85ad441" (UID: "f4feab4f-fead-4471-97da-6393e85ad441"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:21:25.859903 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859897    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-trusted-ca-bundle\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:25.859995 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859910    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-oauth-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:25.859995 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.859921    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:25.861681 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.861657    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f4feab4f-fead-4471-97da-6393e85ad441" (UID: "f4feab4f-fead-4471-97da-6393e85ad441"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:21:25.862036 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.862008    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f4feab4f-fead-4471-97da-6393e85ad441" (UID: "f4feab4f-fead-4471-97da-6393e85ad441"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:21:25.862036 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.862020    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4feab4f-fead-4471-97da-6393e85ad441-kube-api-access-9jvwn" (OuterVolumeSpecName: "kube-api-access-9jvwn") pod "f4feab4f-fead-4471-97da-6393e85ad441" (UID: "f4feab4f-fead-4471-97da-6393e85ad441"). InnerVolumeSpecName "kube-api-access-9jvwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:21:25.960945 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.960836    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-oauth-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:25.960945 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.960886    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4feab4f-fead-4471-97da-6393e85ad441-console-serving-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:25.960945 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.960920    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4feab4f-fead-4471-97da-6393e85ad441-console-config\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:25.960945 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:25.960929    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvwn\" (UniqueName: \"kubernetes.io/projected/f4feab4f-fead-4471-97da-6393e85ad441-kube-api-access-9jvwn\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:26.530060 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:26.530024    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4feab4f-fead-4471-97da-6393e85ad441" containerID="b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a" exitCode=0
Sep 18 14:21:26.530060 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:26.530071    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f7bbbb964-hx9pt"
Sep 18 14:21:26.530639 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:26.530162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7bbbb964-hx9pt" event={"ID":"f4feab4f-fead-4471-97da-6393e85ad441","Type":"ContainerDied","Data":"b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a"}
Sep 18 14:21:26.530639 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:26.530215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f7bbbb964-hx9pt" event={"ID":"f4feab4f-fead-4471-97da-6393e85ad441","Type":"ContainerDied","Data":"b06260c0d1119e545629ebb98204d97a33acbcd71aba4efccaa092971c1a0f4a"}
Sep 18 14:21:26.530639 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:26.530238    2563 scope.go:117] "RemoveContainer" containerID="b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a"
Sep 18 14:21:26.542000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:26.541974    2563 scope.go:117] "RemoveContainer" containerID="b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a"
Sep 18 14:21:26.542358 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:21:26.542337    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a\": container with ID starting with b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a not found: ID does not exist" containerID="b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a"
Sep 18 14:21:26.542446 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:26.542372    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a"} err="failed to get container status \"b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a\": rpc error: code = NotFound desc = could not find container \"b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a\": container with ID starting with b2b4a9bd4297a8681fb0e0842438ee657fd905ac593de10cb5c409c3e0ed225a not found: ID does not exist"
Sep 18 14:21:26.551483 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:26.551396    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f7bbbb964-hx9pt"]
Sep 18 14:21:26.555182 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:26.555150    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f7bbbb964-hx9pt"]
Sep 18 14:21:27.029498 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.029452    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff" podUID="69fc84eb-4f27-4f00-8363-da99ad504306" containerName="main" containerID="cri-o://c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d" gracePeriod=45
Sep 18 14:21:27.300744 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.300712    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:21:27.476872 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.476825    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-tmp-dir\") pod \"69fc84eb-4f27-4f00-8363-da99ad504306\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") "
Sep 18 14:21:27.477059 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.476886    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fc84eb-4f27-4f00-8363-da99ad504306-tls-certs\") pod \"69fc84eb-4f27-4f00-8363-da99ad504306\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") "
Sep 18 14:21:27.477059 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.476922    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2k29\" (UniqueName: \"kubernetes.io/projected/69fc84eb-4f27-4f00-8363-da99ad504306-kube-api-access-m2k29\") pod \"69fc84eb-4f27-4f00-8363-da99ad504306\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") "
Sep 18 14:21:27.477059 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.476984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-dshm\") pod \"69fc84eb-4f27-4f00-8363-da99ad504306\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") "
Sep 18 14:21:27.477229 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.477187    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-home\") pod \"69fc84eb-4f27-4f00-8363-da99ad504306\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") "
Sep 18 14:21:27.477283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.477238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-model-cache\") pod \"69fc84eb-4f27-4f00-8363-da99ad504306\" (UID: \"69fc84eb-4f27-4f00-8363-da99ad504306\") "
Sep 18 14:21:27.477283 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.477247    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69fc84eb-4f27-4f00-8363-da99ad504306" (UID: "69fc84eb-4f27-4f00-8363-da99ad504306"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:21:27.477541 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.477493    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-home" (OuterVolumeSpecName: "home") pod "69fc84eb-4f27-4f00-8363-da99ad504306" (UID: "69fc84eb-4f27-4f00-8363-da99ad504306"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:21:27.477632 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.477539    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-model-cache" (OuterVolumeSpecName: "model-cache") pod "69fc84eb-4f27-4f00-8363-da99ad504306" (UID: "69fc84eb-4f27-4f00-8363-da99ad504306"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:21:27.477747 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.477729    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:27.477804 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.477753    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:27.477804 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.477769    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:27.479398 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.479377    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fc84eb-4f27-4f00-8363-da99ad504306-kube-api-access-m2k29" (OuterVolumeSpecName: "kube-api-access-m2k29") pod "69fc84eb-4f27-4f00-8363-da99ad504306" (UID: "69fc84eb-4f27-4f00-8363-da99ad504306"). InnerVolumeSpecName "kube-api-access-m2k29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:21:27.479723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.479704    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fc84eb-4f27-4f00-8363-da99ad504306-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69fc84eb-4f27-4f00-8363-da99ad504306" (UID: "69fc84eb-4f27-4f00-8363-da99ad504306"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:21:27.479723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.479713    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-dshm" (OuterVolumeSpecName: "dshm") pod "69fc84eb-4f27-4f00-8363-da99ad504306" (UID: "69fc84eb-4f27-4f00-8363-da99ad504306"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:21:27.538349 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.538249    2563 generic.go:356] "Generic (PLEG): container finished" podID="69fc84eb-4f27-4f00-8363-da99ad504306" containerID="c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d" exitCode=0
Sep 18 14:21:27.538349 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.538296    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"
Sep 18 14:21:27.538349 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.538335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff" event={"ID":"69fc84eb-4f27-4f00-8363-da99ad504306","Type":"ContainerDied","Data":"c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d"}
Sep 18 14:21:27.538900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.538385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff" event={"ID":"69fc84eb-4f27-4f00-8363-da99ad504306","Type":"ContainerDied","Data":"beb7bee62909b1012d7aa94e0a593060be5c5dacf67693046462dae0207e4560"}
Sep 18 14:21:27.538900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.538407    2563 scope.go:117] "RemoveContainer" containerID="c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d"
Sep 18 14:21:27.548210 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.548189    2563 scope.go:117] "RemoveContainer" containerID="c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d"
Sep 18 14:21:27.548541 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:21:27.548518    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d\": container with ID starting with c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d not found: ID does not exist" containerID="c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d"
Sep 18 14:21:27.548591 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.548551    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d"} err="failed to get container status \"c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d\": rpc error: code = NotFound desc = could not find container \"c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d\": container with ID starting with c8d4890eeee53f87d3b56f032eee21e907675471b6351db29af15850a249284d not found: ID does not exist"
Sep 18 14:21:27.567864 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.567819    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"]
Sep 18 14:21:27.570989 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.570954    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-988c9d38/e2e-lifecycle-model-988c9d38-kserve-547bccf7cf-pnlff"]
Sep 18 14:21:27.578766 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.578734    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2k29\" (UniqueName: \"kubernetes.io/projected/69fc84eb-4f27-4f00-8363-da99ad504306-kube-api-access-m2k29\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:27.578766 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.578766    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fc84eb-4f27-4f00-8363-da99ad504306-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:27.578907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:27.578777    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fc84eb-4f27-4f00-8363-da99ad504306-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:21:28.288957 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:28.288924    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69fc84eb-4f27-4f00-8363-da99ad504306" path="/var/lib/kubelet/pods/69fc84eb-4f27-4f00-8363-da99ad504306/volumes"
Sep 18 14:21:28.289374 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:28.289359    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4feab4f-fead-4471-97da-6393e85ad441" path="/var/lib/kubelet/pods/f4feab4f-fead-4471-97da-6393e85ad441/volumes"
Sep 18 14:21:32.529793 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:32.529756    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:21:32.538933 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:32.538897    2563 scope.go:117] "RemoveContainer" containerID="19e8f5d6d5060826cb4ab7ede0e6c37c5de4ab574a4d045c701d4585890481a4"
Sep 18 14:21:39.558534 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.558496    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"]
Sep 18 14:21:39.558932 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.558904    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69fc84eb-4f27-4f00-8363-da99ad504306" containerName="main"
Sep 18 14:21:39.558932 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.558915    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69fc84eb-4f27-4f00-8363-da99ad504306" containerName="main"
Sep 18 14:21:39.558932 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.558925    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4feab4f-fead-4471-97da-6393e85ad441" containerName="console"
Sep 18 14:21:39.558932 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.558931    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4feab4f-fead-4471-97da-6393e85ad441" containerName="console"
Sep 18 14:21:39.559103 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.559006    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4feab4f-fead-4471-97da-6393e85ad441" containerName="console"
Sep 18 14:21:39.559103 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.559018    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69fc84eb-4f27-4f00-8363-da99ad504306" containerName="main"
Sep 18 14:21:39.563192 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.563173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.567501 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.567478    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-980661\"/\"auth-test-model-980661-kserve-self-signed-certs\""
Sep 18 14:21:39.567684 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.567532    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-980661\"/\"default-dockercfg-qvxhp\""
Sep 18 14:21:39.567684 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.567572    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-980661\"/\"openshift-service-ca.crt\""
Sep 18 14:21:39.567779 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.567686    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-980661\"/\"kube-root-ca.crt\""
Sep 18 14:21:39.590752 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.590722    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"]
Sep 18 14:21:39.602105 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.602070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-model-cache\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.602281 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.602173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69235290-0a32-4d84-978e-30118c9285d9-tls-certs\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.602281 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.602220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-dshm\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.602281 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.602248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79l4r\" (UniqueName: \"kubernetes.io/projected/69235290-0a32-4d84-978e-30118c9285d9-kube-api-access-79l4r\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.602281 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.602269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-home\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.602427 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.602293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-tmp-dir\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.703789 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.703744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-dshm\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.703990 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.703805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79l4r\" (UniqueName: \"kubernetes.io/projected/69235290-0a32-4d84-978e-30118c9285d9-kube-api-access-79l4r\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.703990 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.703872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-home\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.703990 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.703912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-tmp-dir\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.704209 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.703993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-model-cache\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.704209 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.704046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69235290-0a32-4d84-978e-30118c9285d9-tls-certs\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.704399 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.704379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-home\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.704454 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.704412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-tmp-dir\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.704493 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.704456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-model-cache\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.706091 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.706075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-dshm\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.706598 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.706580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69235290-0a32-4d84-978e-30118c9285d9-tls-certs\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.722878 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.722844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79l4r\" (UniqueName: \"kubernetes.io/projected/69235290-0a32-4d84-978e-30118c9285d9-kube-api-access-79l4r\") pod \"auth-test-model-980661-kserve-86f7bc8dc5-b4b87\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") " pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:39.872326 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:39.872229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:40.147720 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:40.147687    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"]
Sep 18 14:21:40.148913 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:21:40.148886    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69235290_0a32_4d84_978e_30118c9285d9.slice/crio-d91104c489c585151cb53e1178416c86fc27a5ec83d04ee347dbf61d685126de WatchSource:0}: Error finding container d91104c489c585151cb53e1178416c86fc27a5ec83d04ee347dbf61d685126de: Status 404 returned error can't find the container with id d91104c489c585151cb53e1178416c86fc27a5ec83d04ee347dbf61d685126de
Sep 18 14:21:40.628138 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:40.628102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87" event={"ID":"69235290-0a32-4d84-978e-30118c9285d9","Type":"ContainerStarted","Data":"0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de"}
Sep 18 14:21:40.628536 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:40.628177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87" event={"ID":"69235290-0a32-4d84-978e-30118c9285d9","Type":"ContainerStarted","Data":"d91104c489c585151cb53e1178416c86fc27a5ec83d04ee347dbf61d685126de"}
Sep 18 14:21:40.647757 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:40.647702    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87" podStartSLOduration=1.6476832890000002 podStartE2EDuration="1.647683289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:21:40.644177668 +0000 UTC m=+1388.800697946" watchObservedRunningTime="2026-09-18 14:21:40.647683289 +0000 UTC m=+1388.804203570"
Sep 18 14:21:49.872370 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:49.872324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:49.872370 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:49.872373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:49.885466 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:49.885437    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:21:50.698410 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:21:50.698369    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:22:02.703782 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:02.703749    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"]
Sep 18 14:22:02.704375 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:02.704017    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy" containerID="cri-o://e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9" gracePeriod=30
Sep 18 14:22:07.961113 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:07.961078    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:22:08.104073 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.103978    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-data\") pod \"dd4f751a-c79c-4c15-b24c-d508367dd676\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") "
Sep 18 14:22:08.104073 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-envoy\") pod \"dd4f751a-c79c-4c15-b24c-d508367dd676\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") "
Sep 18 14:22:08.104073 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104044    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wxlw\" (UniqueName: \"kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-kube-api-access-8wxlw\") pod \"dd4f751a-c79c-4c15-b24c-d508367dd676\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") "
Sep 18 14:22:08.104073 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104075    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-credential-socket\") pod \"dd4f751a-c79c-4c15-b24c-d508367dd676\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") "
Sep 18 14:22:08.104435 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104099    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4f751a-c79c-4c15-b24c-d508367dd676-istiod-ca-cert\") pod \"dd4f751a-c79c-4c15-b24c-d508367dd676\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") "
Sep 18 14:22:08.104435 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104303    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-podinfo\") pod \"dd4f751a-c79c-4c15-b24c-d508367dd676\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") "
Sep 18 14:22:08.104435 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104365    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd4f751a-c79c-4c15-b24c-d508367dd676" (UID: "dd4f751a-c79c-4c15-b24c-d508367dd676"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:08.104435 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-socket\") pod \"dd4f751a-c79c-4c15-b24c-d508367dd676\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") "
Sep 18 14:22:08.104648 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-certs\") pod \"dd4f751a-c79c-4c15-b24c-d508367dd676\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") "
Sep 18 14:22:08.104648 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104508    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd4f751a-c79c-4c15-b24c-d508367dd676" (UID: "dd4f751a-c79c-4c15-b24c-d508367dd676"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:08.104648 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104575    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd4f751a-c79c-4c15-b24c-d508367dd676" (UID: "dd4f751a-c79c-4c15-b24c-d508367dd676"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:08.104648 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-token\") pod \"dd4f751a-c79c-4c15-b24c-d508367dd676\" (UID: \"dd4f751a-c79c-4c15-b24c-d508367dd676\") "
Sep 18 14:22:08.104880 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104758    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd4f751a-c79c-4c15-b24c-d508367dd676" (UID: "dd4f751a-c79c-4c15-b24c-d508367dd676"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:08.104880 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104785    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4f751a-c79c-4c15-b24c-d508367dd676-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd4f751a-c79c-4c15-b24c-d508367dd676" (UID: "dd4f751a-c79c-4c15-b24c-d508367dd676"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:22:08.105008 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104974    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:22:08.105008 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104988    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-workload-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:22:08.105008 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.104997    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-data\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:22:08.105008 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.105005    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-credential-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:22:08.105254 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.105013    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4f751a-c79c-4c15-b24c-d508367dd676-istiod-ca-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:22:08.106507 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.106480    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd4f751a-c79c-4c15-b24c-d508367dd676" (UID: "dd4f751a-c79c-4c15-b24c-d508367dd676"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:22:08.106834 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.106808    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd4f751a-c79c-4c15-b24c-d508367dd676" (UID: "dd4f751a-c79c-4c15-b24c-d508367dd676"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:22:08.106926 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.106835    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-kube-api-access-8wxlw" (OuterVolumeSpecName: "kube-api-access-8wxlw") pod "dd4f751a-c79c-4c15-b24c-d508367dd676" (UID: "dd4f751a-c79c-4c15-b24c-d508367dd676"). InnerVolumeSpecName "kube-api-access-8wxlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:22:08.106926 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.106834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd4f751a-c79c-4c15-b24c-d508367dd676" (UID: "dd4f751a-c79c-4c15-b24c-d508367dd676"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:22:08.206398 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.206359    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-token\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:22:08.206398 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.206389    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-envoy\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:22:08.206398 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.206398    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wxlw\" (UniqueName: \"kubernetes.io/projected/dd4f751a-c79c-4c15-b24c-d508367dd676-kube-api-access-8wxlw\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:22:08.206398 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.206407    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4f751a-c79c-4c15-b24c-d508367dd676-istio-podinfo\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:22:08.800467 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.800426    2563 generic.go:356] "Generic (PLEG): container finished" podID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerID="e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9" exitCode=0
Sep 18 14:22:08.800651 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.800483    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"
Sep 18 14:22:08.800651 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.800515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" event={"ID":"dd4f751a-c79c-4c15-b24c-d508367dd676","Type":"ContainerDied","Data":"e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9"}
Sep 18 14:22:08.800651 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.800571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k" event={"ID":"dd4f751a-c79c-4c15-b24c-d508367dd676","Type":"ContainerDied","Data":"faae42fa9ba8e2e0bd2b5deec8f7e0540cb487dcb76eea5257e4307e51d5ed54"}
Sep 18 14:22:08.800651 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.800597    2563 scope.go:117] "RemoveContainer" containerID="e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9"
Sep 18 14:22:08.810929 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.810912    2563 scope.go:117] "RemoveContainer" containerID="f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7"
Sep 18 14:22:08.819335 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.819303    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"]
Sep 18 14:22:08.820524 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.820399    2563 scope.go:117] "RemoveContainer" containerID="e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9"
Sep 18 14:22:08.820801 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:22:08.820772    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9\": container with ID starting with e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9 not found: ID does not exist" containerID="e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9"
Sep 18 14:22:08.820909 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.820812    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9"} err="failed to get container status \"e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9\": rpc error: code = NotFound desc = could not find container \"e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9\": container with ID starting with e783473a8c27f51d7cff451ac68b7746ff94bc600dc8f8783ee6242ba11166d9 not found: ID does not exist"
Sep 18 14:22:08.820909 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.820834    2563 scope.go:117] "RemoveContainer" containerID="f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7"
Sep 18 14:22:08.821458 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:22:08.821429    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7\": container with ID starting with f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7 not found: ID does not exist" containerID="f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7"
Sep 18 14:22:08.821549 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.821466    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7"} err="failed to get container status \"f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7\": rpc error: code = NotFound desc = could not find container \"f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7\": container with ID starting with f4457bf934cc82ff8afe188d6a2ef89477ea26c71a47e1a14bed54da3852f1f7 not found: ID does not exist"
Sep 18 14:22:08.823688 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:08.823667    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-988c9d38-openshift-default-79975f7f6d-99f4k"]
Sep 18 14:22:10.288881 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:10.288841    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" path="/var/lib/kubelet/pods/dd4f751a-c79c-4c15-b24c-d508367dd676/volumes"
Sep 18 14:22:11.249589 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:11.249546    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:22:45.859705 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:45.859665    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"]
Sep 18 14:22:45.886137 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:45.886087    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"]
Sep 18 14:22:45.896847 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:45.896815    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"]
Sep 18 14:22:49.622689 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:49.622646    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:22:56.900476 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:56.900433    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss"]
Sep 18 14:22:56.906603 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:56.906567    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4d9222-29829015-cz9ss"]
Sep 18 14:22:56.953718 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:56.953681    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"]
Sep 18 14:22:56.953968 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:56.953918    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx" podUID="dcaf20b7-f9d7-45bd-8dd1-57d79812916f" containerName="payload-processing" containerID="cri-o://b1689ca5711d03e871d6a080ac14e8db1c82a83035bd491e194f44ba52b5fb1c" gracePeriod=30
Sep 18 14:22:57.101502 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:57.101468    2563 generic.go:356] "Generic (PLEG): container finished" podID="dcaf20b7-f9d7-45bd-8dd1-57d79812916f" containerID="b1689ca5711d03e871d6a080ac14e8db1c82a83035bd491e194f44ba52b5fb1c" exitCode=0
Sep 18 14:22:57.101671 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:57.101544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx" event={"ID":"dcaf20b7-f9d7-45bd-8dd1-57d79812916f","Type":"ContainerDied","Data":"b1689ca5711d03e871d6a080ac14e8db1c82a83035bd491e194f44ba52b5fb1c"}
Sep 18 14:22:57.207503 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:57.207474    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:22:57.310182 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:57.310143    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-plugins-config-volume\") pod \"dcaf20b7-f9d7-45bd-8dd1-57d79812916f\" (UID: \"dcaf20b7-f9d7-45bd-8dd1-57d79812916f\") "
Sep 18 14:22:57.310376 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:57.310216    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mbh8\" (UniqueName: \"kubernetes.io/projected/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-kube-api-access-6mbh8\") pod \"dcaf20b7-f9d7-45bd-8dd1-57d79812916f\" (UID: \"dcaf20b7-f9d7-45bd-8dd1-57d79812916f\") "
Sep 18 14:22:57.310508 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:57.310484    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dcaf20b7-f9d7-45bd-8dd1-57d79812916f" (UID: "dcaf20b7-f9d7-45bd-8dd1-57d79812916f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:22:57.310589 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:57.310572    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-plugins-config-volume\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:22:57.312317 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:57.312286    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-kube-api-access-6mbh8" (OuterVolumeSpecName: "kube-api-access-6mbh8") pod "dcaf20b7-f9d7-45bd-8dd1-57d79812916f" (UID: "dcaf20b7-f9d7-45bd-8dd1-57d79812916f"). InnerVolumeSpecName "kube-api-access-6mbh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:22:57.411532 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:57.411490    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mbh8\" (UniqueName: \"kubernetes.io/projected/dcaf20b7-f9d7-45bd-8dd1-57d79812916f-kube-api-access-6mbh8\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:22:58.108435 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:58.108400    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"
Sep 18 14:22:58.108871 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:58.108431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx" event={"ID":"dcaf20b7-f9d7-45bd-8dd1-57d79812916f","Type":"ContainerDied","Data":"7a856324bc56e47e996cf26196a35488893a142dcecdaae8125a9b53b1f1250b"}
Sep 18 14:22:58.108871 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:58.108496    2563 scope.go:117] "RemoveContainer" containerID="b1689ca5711d03e871d6a080ac14e8db1c82a83035bd491e194f44ba52b5fb1c"
Sep 18 14:22:58.160146 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:58.160095    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"]
Sep 18 14:22:58.195924 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:58.195887    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4d9222-6f754b67b5-d8grx"]
Sep 18 14:22:58.288585 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:58.288549    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="650bc3c6-851e-4c11-95b6-30ea3a0f3c64" path="/var/lib/kubelet/pods/650bc3c6-851e-4c11-95b6-30ea3a0f3c64/volumes"
Sep 18 14:22:58.288931 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:22:58.288912    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcaf20b7-f9d7-45bd-8dd1-57d79812916f" path="/var/lib/kubelet/pods/dcaf20b7-f9d7-45bd-8dd1-57d79812916f/volumes"
Sep 18 14:23:00.879614 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:00.879536    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd" podUID="b3ac69e0-a9c9-4c14-a717-73854425340a" containerName="main" containerID="cri-o://b70dd65d2c9abfa6b00d4b215a3fe2e8409b774e41cc7516640802fe090abb35" gracePeriod=45
Sep 18 14:23:00.906101 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:00.906064    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx" podUID="0c78903b-eb41-4fe3-a8f3-98a48da8d959" containerName="main" containerID="cri-o://eb69cfae152d0ec359e5aa0eabfcacadaf39641ceb71474d81f5966e34ae73cd" gracePeriod=45
Sep 18 14:23:00.916103 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:00.916039    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6" podUID="24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" containerName="main" containerID="cri-o://65b4fe15951eda56bb485cd081b9cec197452528cf8c31d77d78eeae62b23b89" gracePeriod=45
Sep 18 14:23:01.131789 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.131442    2563 generic.go:356] "Generic (PLEG): container finished" podID="b3ac69e0-a9c9-4c14-a717-73854425340a" containerID="b70dd65d2c9abfa6b00d4b215a3fe2e8409b774e41cc7516640802fe090abb35" exitCode=0
Sep 18 14:23:01.131789 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.131688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd" event={"ID":"b3ac69e0-a9c9-4c14-a717-73854425340a","Type":"ContainerDied","Data":"b70dd65d2c9abfa6b00d4b215a3fe2e8409b774e41cc7516640802fe090abb35"}
Sep 18 14:23:01.134855 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.134712    2563 generic.go:356] "Generic (PLEG): container finished" podID="24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" containerID="65b4fe15951eda56bb485cd081b9cec197452528cf8c31d77d78eeae62b23b89" exitCode=0
Sep 18 14:23:01.134855 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.134813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6" event={"ID":"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0","Type":"ContainerDied","Data":"65b4fe15951eda56bb485cd081b9cec197452528cf8c31d77d78eeae62b23b89"}
Sep 18 14:23:01.137069 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.136963    2563 generic.go:356] "Generic (PLEG): container finished" podID="0c78903b-eb41-4fe3-a8f3-98a48da8d959" containerID="eb69cfae152d0ec359e5aa0eabfcacadaf39641ceb71474d81f5966e34ae73cd" exitCode=0
Sep 18 14:23:01.137069 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.137022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx" event={"ID":"0c78903b-eb41-4fe3-a8f3-98a48da8d959","Type":"ContainerDied","Data":"eb69cfae152d0ec359e5aa0eabfcacadaf39641ceb71474d81f5966e34ae73cd"}
Sep 18 14:23:01.222735 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.222707    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:23:01.226065 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.226036    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:23:01.230311 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.230285    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:23:01.355501 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrbk8\" (UniqueName: \"kubernetes.io/projected/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-kube-api-access-rrbk8\") pod \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") "
Sep 18 14:23:01.355691 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355512    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tmp-dir\") pod \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") "
Sep 18 14:23:01.355691 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tls-certs\") pod \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") "
Sep 18 14:23:01.355691 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355596    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tls-certs\") pod \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") "
Sep 18 14:23:01.355691 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355667    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-model-cache\") pod \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") "
Sep 18 14:23:01.355691 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-tmp-dir\") pod \"b3ac69e0-a9c9-4c14-a717-73854425340a\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") "
Sep 18 14:23:01.355960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355716    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ac69e0-a9c9-4c14-a717-73854425340a-tls-certs\") pod \"b3ac69e0-a9c9-4c14-a717-73854425340a\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") "
Sep 18 14:23:01.355960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355747    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-model-cache\") pod \"b3ac69e0-a9c9-4c14-a717-73854425340a\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") "
Sep 18 14:23:01.355960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tmp-dir\") pod \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") "
Sep 18 14:23:01.355960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355808    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-dshm\") pod \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") "
Sep 18 14:23:01.355960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355832    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-dshm\") pod \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") "
Sep 18 14:23:01.355960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355856    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-home\") pod \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") "
Sep 18 14:23:01.355960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355875    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c78903b-eb41-4fe3-a8f3-98a48da8d959" (UID: "0c78903b-eb41-4fe3-a8f3-98a48da8d959"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.355960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c78903b-eb41-4fe3-a8f3-98a48da8d959" (UID: "0c78903b-eb41-4fe3-a8f3-98a48da8d959"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.355960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355880    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-home\") pod \"b3ac69e0-a9c9-4c14-a717-73854425340a\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") "
Sep 18 14:23:01.355960 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355913    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3ac69e0-a9c9-4c14-a717-73854425340a" (UID: "b3ac69e0-a9c9-4c14-a717-73854425340a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.356494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.355991    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-dshm\") pod \"b3ac69e0-a9c9-4c14-a717-73854425340a\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") "
Sep 18 14:23:01.356494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-home\") pod \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") "
Sep 18 14:23:01.356494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356053    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-model-cache\") pod \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\" (UID: \"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0\") "
Sep 18 14:23:01.356494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356063    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" (UID: "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.356494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356093    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-home" (OuterVolumeSpecName: "home") pod "b3ac69e0-a9c9-4c14-a717-73854425340a" (UID: "b3ac69e0-a9c9-4c14-a717-73854425340a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.356494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qprxw\" (UniqueName: \"kubernetes.io/projected/b3ac69e0-a9c9-4c14-a717-73854425340a-kube-api-access-qprxw\") pod \"b3ac69e0-a9c9-4c14-a717-73854425340a\" (UID: \"b3ac69e0-a9c9-4c14-a717-73854425340a\") "
Sep 18 14:23:01.356494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxfcw\" (UniqueName: \"kubernetes.io/projected/0c78903b-eb41-4fe3-a8f3-98a48da8d959-kube-api-access-zxfcw\") pod \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\" (UID: \"0c78903b-eb41-4fe3-a8f3-98a48da8d959\") "
Sep 18 14:23:01.356494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356175    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3ac69e0-a9c9-4c14-a717-73854425340a" (UID: "b3ac69e0-a9c9-4c14-a717-73854425340a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.356494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356222    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" (UID: "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.356976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356575    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.356976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356598    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.356976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356611    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.356976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356624    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.356976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356636    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.356976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356648    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.356976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.356659    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.357946 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.357680    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-home" (OuterVolumeSpecName: "home") pod "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" (UID: "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.357946 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.357696    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-home" (OuterVolumeSpecName: "home") pod "0c78903b-eb41-4fe3-a8f3-98a48da8d959" (UID: "0c78903b-eb41-4fe3-a8f3-98a48da8d959"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.358540 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.358518    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-dshm" (OuterVolumeSpecName: "dshm") pod "0c78903b-eb41-4fe3-a8f3-98a48da8d959" (UID: "0c78903b-eb41-4fe3-a8f3-98a48da8d959"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.358639 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.358527    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-kube-api-access-rrbk8" (OuterVolumeSpecName: "kube-api-access-rrbk8") pod "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" (UID: "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0"). InnerVolumeSpecName "kube-api-access-rrbk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:23:01.358752 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.358704    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c78903b-eb41-4fe3-a8f3-98a48da8d959" (UID: "0c78903b-eb41-4fe3-a8f3-98a48da8d959"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:01.358916 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.358894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" (UID: "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:01.359009 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.358993    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c78903b-eb41-4fe3-a8f3-98a48da8d959-kube-api-access-zxfcw" (OuterVolumeSpecName: "kube-api-access-zxfcw") pod "0c78903b-eb41-4fe3-a8f3-98a48da8d959" (UID: "0c78903b-eb41-4fe3-a8f3-98a48da8d959"). InnerVolumeSpecName "kube-api-access-zxfcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:23:01.359115 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.359088    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-dshm" (OuterVolumeSpecName: "dshm") pod "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" (UID: "24aa8ad7-fee6-458b-bbb6-a6d48d697bc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.359474 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.359460    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-dshm" (OuterVolumeSpecName: "dshm") pod "b3ac69e0-a9c9-4c14-a717-73854425340a" (UID: "b3ac69e0-a9c9-4c14-a717-73854425340a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:01.359941 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.359923    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ac69e0-a9c9-4c14-a717-73854425340a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3ac69e0-a9c9-4c14-a717-73854425340a" (UID: "b3ac69e0-a9c9-4c14-a717-73854425340a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:01.360119 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.360099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ac69e0-a9c9-4c14-a717-73854425340a-kube-api-access-qprxw" (OuterVolumeSpecName: "kube-api-access-qprxw") pod "b3ac69e0-a9c9-4c14-a717-73854425340a" (UID: "b3ac69e0-a9c9-4c14-a717-73854425340a"). InnerVolumeSpecName "kube-api-access-qprxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:23:01.457204 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.457169    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3ac69e0-a9c9-4c14-a717-73854425340a-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.457204 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.457199    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.457204 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.457209    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qprxw\" (UniqueName: \"kubernetes.io/projected/b3ac69e0-a9c9-4c14-a717-73854425340a-kube-api-access-qprxw\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.457440 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.457220    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxfcw\" (UniqueName: \"kubernetes.io/projected/0c78903b-eb41-4fe3-a8f3-98a48da8d959-kube-api-access-zxfcw\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.457440 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.457229    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrbk8\" (UniqueName: \"kubernetes.io/projected/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-kube-api-access-rrbk8\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.457440 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.457237    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.457440 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.457245    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c78903b-eb41-4fe3-a8f3-98a48da8d959-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.457440 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.457253    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ac69e0-a9c9-4c14-a717-73854425340a-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.457440 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.457259    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c78903b-eb41-4fe3-a8f3-98a48da8d959-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.457440 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.457266    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:01.457440 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:01.457272    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:02.144981 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.144949    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"
Sep 18 14:23:02.145468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.144982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd" event={"ID":"b3ac69e0-a9c9-4c14-a717-73854425340a","Type":"ContainerDied","Data":"4a06fc6cef7dc40a23fa20aa4a85e39bd8bb6d8a826a43a37a2a2b59d732155f"}
Sep 18 14:23:02.145468 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.145029    2563 scope.go:117] "RemoveContainer" containerID="b70dd65d2c9abfa6b00d4b215a3fe2e8409b774e41cc7516640802fe090abb35"
Sep 18 14:23:02.146685 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.146617    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"
Sep 18 14:23:02.146685 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.146654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6" event={"ID":"24aa8ad7-fee6-458b-bbb6-a6d48d697bc0","Type":"ContainerDied","Data":"cc5bc010a371d392dd74488c76d27087b8f1db36ef265b9d342f4126b0c6fb0b"}
Sep 18 14:23:02.148180 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.148154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx" event={"ID":"0c78903b-eb41-4fe3-a8f3-98a48da8d959","Type":"ContainerDied","Data":"53f3595487092b2483d6a1cfd29d970cd94e070d70b4c7fc6ea3cc5b79172d39"}
Sep 18 14:23:02.148272 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.148162    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"
Sep 18 14:23:02.155228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.155210    2563 scope.go:117] "RemoveContainer" containerID="65b4fe15951eda56bb485cd081b9cec197452528cf8c31d77d78eeae62b23b89"
Sep 18 14:23:02.164921 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.164904    2563 scope.go:117] "RemoveContainer" containerID="eb69cfae152d0ec359e5aa0eabfcacadaf39641ceb71474d81f5966e34ae73cd"
Sep 18 14:23:02.183455 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.183416    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"]
Sep 18 14:23:02.191966 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.191938    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/distinct-2-w0-4d9222-kserve-55584545f5-ngkjd"]
Sep 18 14:23:02.216419 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.216384    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"]
Sep 18 14:23:02.219928 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.219899    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/premium-w0-4d9222-kserve-5dd89f6577-wdvzx"]
Sep 18 14:23:02.231327 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.231297    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"]
Sep 18 14:23:02.237713 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.237686    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4d9222/simulator-w0-4d9222-kserve-78cb7f4f88-29hw6"]
Sep 18 14:23:02.289266 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.289233    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c78903b-eb41-4fe3-a8f3-98a48da8d959" path="/var/lib/kubelet/pods/0c78903b-eb41-4fe3-a8f3-98a48da8d959/volumes"
Sep 18 14:23:02.289659 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.289646    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" path="/var/lib/kubelet/pods/24aa8ad7-fee6-458b-bbb6-a6d48d697bc0/volumes"
Sep 18 14:23:02.290010 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:02.289998    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3ac69e0-a9c9-4c14-a717-73854425340a" path="/var/lib/kubelet/pods/b3ac69e0-a9c9-4c14-a717-73854425340a/volumes"
Sep 18 14:23:08.097317 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097280    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"]
Sep 18 14:23:08.097703 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097687    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy"
Sep 18 14:23:08.097742 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097709    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy"
Sep 18 14:23:08.097742 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097721    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ac69e0-a9c9-4c14-a717-73854425340a" containerName="main"
Sep 18 14:23:08.097742 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097728    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ac69e0-a9c9-4c14-a717-73854425340a" containerName="main"
Sep 18 14:23:08.097742 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097734    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy"
Sep 18 14:23:08.097742 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097740    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy"
Sep 18 14:23:08.097907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097749    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcaf20b7-f9d7-45bd-8dd1-57d79812916f" containerName="payload-processing"
Sep 18 14:23:08.097907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097755    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcaf20b7-f9d7-45bd-8dd1-57d79812916f" containerName="payload-processing"
Sep 18 14:23:08.097907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097765    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c78903b-eb41-4fe3-a8f3-98a48da8d959" containerName="main"
Sep 18 14:23:08.097907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097774    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c78903b-eb41-4fe3-a8f3-98a48da8d959" containerName="main"
Sep 18 14:23:08.097907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097791    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" containerName="main"
Sep 18 14:23:08.097907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097797    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" containerName="main"
Sep 18 14:23:08.097907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097866    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ac69e0-a9c9-4c14-a717-73854425340a" containerName="main"
Sep 18 14:23:08.097907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097876    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy"
Sep 18 14:23:08.097907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097882    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcaf20b7-f9d7-45bd-8dd1-57d79812916f" containerName="payload-processing"
Sep 18 14:23:08.097907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097887    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c78903b-eb41-4fe3-a8f3-98a48da8d959" containerName="main"
Sep 18 14:23:08.097907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.097892    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="24aa8ad7-fee6-458b-bbb6-a6d48d697bc0" containerName="main"
Sep 18 14:23:08.101138 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.101105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.104380 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.104358    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e01050-serving-cert\""
Sep 18 14:23:08.113988 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.113962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"]
Sep 18 14:23:08.227174 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.227109    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwg28\" (UniqueName: \"kubernetes.io/projected/f2bfae85-d825-4107-a53b-c1e0084a29ce-kube-api-access-jwg28\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.227174 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.227178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-metrics-certs\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.227420 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.227213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-maas-api-tls\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.227420 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.227232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bfae85-d825-4107-a53b-c1e0084a29ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.328729 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.328688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-metrics-certs\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.328930 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.328746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-maas-api-tls\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.328930 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.328773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bfae85-d825-4107-a53b-c1e0084a29ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.328930 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.328898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwg28\" (UniqueName: \"kubernetes.io/projected/f2bfae85-d825-4107-a53b-c1e0084a29ce-kube-api-access-jwg28\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.329523 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.329494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bfae85-d825-4107-a53b-c1e0084a29ce-openshift-service-ca\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.331410 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.331385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-maas-api-tls\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.331524 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.331386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-metrics-certs\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.338172 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.338145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwg28\" (UniqueName: \"kubernetes.io/projected/f2bfae85-d825-4107-a53b-c1e0084a29ce-kube-api-access-jwg28\") pod \"maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.413740 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.413651    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:08.694255 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.694227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"]
Sep 18 14:23:08.695604 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:23:08.695575    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bfae85_d825_4107_a53b_c1e0084a29ce.slice/crio-3e30a18bcc154bd561e62e57bfda2bd3e3fd2cfb9fa15955d7edf4a413ba0afd WatchSource:0}: Error finding container 3e30a18bcc154bd561e62e57bfda2bd3e3fd2cfb9fa15955d7edf4a413ba0afd: Status 404 returned error can't find the container with id 3e30a18bcc154bd561e62e57bfda2bd3e3fd2cfb9fa15955d7edf4a413ba0afd
Sep 18 14:23:08.697066 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:08.697045    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:23:09.201946 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:09.201917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l" event={"ID":"f2bfae85-d825-4107-a53b-c1e0084a29ce","Type":"ContainerStarted","Data":"3e30a18bcc154bd561e62e57bfda2bd3e3fd2cfb9fa15955d7edf4a413ba0afd"}
Sep 18 14:23:10.210267 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:10.210216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l" event={"ID":"f2bfae85-d825-4107-a53b-c1e0084a29ce","Type":"ContainerStarted","Data":"ce1feade91ca13078422f3f22d29c4d17c887d5431d7ef35e5daa12390eaf80f"}
Sep 18 14:23:10.210680 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:10.210453    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:10.227923 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:10.227868    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l" podStartSLOduration=1.744062641 podStartE2EDuration="2.227851758s" totalImagesPullingTime="483.789117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:23:10.226529907 +0000 UTC m=+1478.383050190" watchObservedRunningTime="2026-09-18 14:23:10.227851758 +0000 UTC m=+1478.384372037"
Sep 18 14:23:16.221893 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:16.221863    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:23:25.383495 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:25.383454    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"]
Sep 18 14:23:25.389221 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:25.389190    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"]
Sep 18 14:23:25.410853 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:25.410816    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"]
Sep 18 14:23:28.508046 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.508011    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"]
Sep 18 14:23:28.508549 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.508532    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd4f751a-c79c-4c15-b24c-d508367dd676" containerName="istio-proxy"
Sep 18 14:23:28.511929 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.511910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.514472 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.514448    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e9388b-openshift-default-dockercfg-7g5xb\""
Sep 18 14:23:28.514851 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.514835    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 14:23:28.527044 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.527012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"]
Sep 18 14:23:28.626475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.626435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-podinfo\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.626475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.626477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-socket\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.626688 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.626493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-credential-socket\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.626688 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.626614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-certs\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.626688 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.626651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgwn\" (UniqueName: \"kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-kube-api-access-rtgwn\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.626688 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.626683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3078f077-d672-44f4-89a0-97c1c5f4f2de-istiod-ca-cert\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.626828 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.626706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-envoy\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.626828 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.626726    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-data\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.626828 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.626763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-token\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.727617 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.727553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-certs\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.727617 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.727611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgwn\" (UniqueName: \"kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-kube-api-access-rtgwn\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.727875 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.727637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3078f077-d672-44f4-89a0-97c1c5f4f2de-istiod-ca-cert\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.727875 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.727655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-envoy\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.727875 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.727676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-data\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.727875 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.727699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-token\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.727875 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.727763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-podinfo\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.727875 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.727810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-socket\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.727875 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.727834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-credential-socket\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.728255 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.728024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-certs\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.728255 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.728234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-data\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.728335 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.728312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-socket\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.728376 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.728334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-credential-socket\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.728425 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.728408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3078f077-d672-44f4-89a0-97c1c5f4f2de-istiod-ca-cert\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.730036 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.730020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-envoy\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.730332 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.730314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-podinfo\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.736306 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.736282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-token\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.736418 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.736369    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgwn\" (UniqueName: \"kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-kube-api-access-rtgwn\") pod \"e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") " pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:28.822819 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:28.822726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:29.113711 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:29.113681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"]
Sep 18 14:23:29.115520 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:23:29.115494    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3078f077_d672_44f4_89a0_97c1c5f4f2de.slice/crio-62e460057ea76a4432775558ba6a9ef6a9130a763872301e90bfe3b3c3434ca6 WatchSource:0}: Error finding container 62e460057ea76a4432775558ba6a9ef6a9130a763872301e90bfe3b3c3434ca6: Status 404 returned error can't find the container with id 62e460057ea76a4432775558ba6a9ef6a9130a763872301e90bfe3b3c3434ca6
Sep 18 14:23:29.117877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:29.117841    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:23:29.117947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:29.117911    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:23:29.117947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:29.117936    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:23:29.344118 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:29.344070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" event={"ID":"3078f077-d672-44f4-89a0-97c1c5f4f2de","Type":"ContainerStarted","Data":"64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a"}
Sep 18 14:23:29.344344 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:29.344160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" event={"ID":"3078f077-d672-44f4-89a0-97c1c5f4f2de","Type":"ContainerStarted","Data":"62e460057ea76a4432775558ba6a9ef6a9130a763872301e90bfe3b3c3434ca6"}
Sep 18 14:23:29.385933 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:29.385817    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" podStartSLOduration=1.385798083 podStartE2EDuration="1.385798083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:23:29.382881446 +0000 UTC m=+1497.539401717" watchObservedRunningTime="2026-09-18 14:23:29.385798083 +0000 UTC m=+1497.542318472"
Sep 18 14:23:29.823505 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:29.823448    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:30.041439 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:30.041401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:23:30.828506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:30.828476    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:31.356213 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:31.356176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:31.357079 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:31.357060    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:23:32.583832 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:32.583796    2563 scope.go:117] "RemoveContainer" containerID="187dbece35a48c4a9855118c39fe8d3681ec53058d0b4b7394731927633eda44"
Sep 18 14:23:32.767391 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:32.767365    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:23:32.778929 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:32.778896    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:23:34.254115 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.254077    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"]
Sep 18 14:23:34.258046 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.258016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.260743 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.260717    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e9388b-serving-cert\""
Sep 18 14:23:34.268529 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.268501    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"]
Sep 18 14:23:34.387958 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.387919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffac570e-c126-450e-8da0-9a9e766ebb0f-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.388173 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.388111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9t8\" (UniqueName: \"kubernetes.io/projected/ffac570e-c126-450e-8da0-9a9e766ebb0f-kube-api-access-5l9t8\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.388173 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.388166    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-maas-api-tls\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.388318 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.388202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-metrics-certs\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.489365 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.489323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9t8\" (UniqueName: \"kubernetes.io/projected/ffac570e-c126-450e-8da0-9a9e766ebb0f-kube-api-access-5l9t8\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.489365 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.489372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-maas-api-tls\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.489638 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.489399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-metrics-certs\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.489638 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.489457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffac570e-c126-450e-8da0-9a9e766ebb0f-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.489638 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:23:34.489508    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-serving-cert: secret "maas-api-e2e-api-b-e9388b-serving-cert" not found
Sep 18 14:23:34.489638 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:23:34.489586    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-maas-api-tls podName:ffac570e-c126-450e-8da0-9a9e766ebb0f nodeName:}" failed. No retries permitted until 2026-09-18 14:23:34.989570374 +0000 UTC m=+1503.146090645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-maas-api-tls") pod "maas-api-e2e-api-b-e9388b-59f554658b-5rnw9" (UID: "ffac570e-c126-450e-8da0-9a9e766ebb0f") : secret "maas-api-e2e-api-b-e9388b-serving-cert" not found
Sep 18 14:23:34.490252 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.490234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffac570e-c126-450e-8da0-9a9e766ebb0f-openshift-service-ca\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.492002 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.491980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-metrics-certs\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.498293 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.498266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9t8\" (UniqueName: \"kubernetes.io/projected/ffac570e-c126-450e-8da0-9a9e766ebb0f-kube-api-access-5l9t8\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:34.995464 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.995422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-maas-api-tls\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:35.005281 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:34.998929    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-maas-api-tls\") pod \"maas-api-e2e-api-b-e9388b-59f554658b-5rnw9\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:35.172566 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:35.172512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:35.450623 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:35.450594    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"]
Sep 18 14:23:35.451884 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:23:35.451853    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffac570e_c126_450e_8da0_9a9e766ebb0f.slice/crio-5be9db67bfd621c9e3f0be71e142030b4ae69282c70ba912e78a29cd637d1869 WatchSource:0}: Error finding container 5be9db67bfd621c9e3f0be71e142030b4ae69282c70ba912e78a29cd637d1869: Status 404 returned error can't find the container with id 5be9db67bfd621c9e3f0be71e142030b4ae69282c70ba912e78a29cd637d1869
Sep 18 14:23:36.389021 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:36.388983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9" event={"ID":"ffac570e-c126-450e-8da0-9a9e766ebb0f","Type":"ContainerStarted","Data":"4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21"}
Sep 18 14:23:36.389260 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:36.389044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:23:36.389260 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:36.389061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9" event={"ID":"ffac570e-c126-450e-8da0-9a9e766ebb0f","Type":"ContainerStarted","Data":"5be9db67bfd621c9e3f0be71e142030b4ae69282c70ba912e78a29cd637d1869"}
Sep 18 14:23:36.409155 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:36.409084    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9" podStartSLOduration=2.039374448 podStartE2EDuration="2.409062665s" totalImagesPullingTime="369.688217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:23:36.406771306 +0000 UTC m=+1504.563291589" watchObservedRunningTime="2026-09-18 14:23:36.409062665 +0000 UTC m=+1504.565582944"
Sep 18 14:23:40.403477 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.403435    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb" podUID="d1311a7e-85fa-48f1-b159-12119da8827d" containerName="main" containerID="cri-o://61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288" gracePeriod=45
Sep 18 14:23:40.407737 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.407688    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks" podUID="4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" containerName="main" containerID="cri-o://e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888" gracePeriod=45
Sep 18 14:23:40.434609 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.434570    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb" podUID="6cccf0cd-2042-4842-a2e6-471225fc7aac" containerName="main" containerID="cri-o://87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6" gracePeriod=45
Sep 18 14:23:40.736271 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.736238    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:23:40.739758 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.739731    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:23:40.754031 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.753998    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:23:40.857588 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-model-cache\") pod \"6cccf0cd-2042-4842-a2e6-471225fc7aac\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") "
Sep 18 14:23:40.857588 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857590    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-dshm\") pod \"d1311a7e-85fa-48f1-b159-12119da8827d\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") "
Sep 18 14:23:40.857900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857621    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-dshm\") pod \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") "
Sep 18 14:23:40.857900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857649    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tls-certs\") pod \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") "
Sep 18 14:23:40.857900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857673    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2dhf\" (UniqueName: \"kubernetes.io/projected/6cccf0cd-2042-4842-a2e6-471225fc7aac-kube-api-access-f2dhf\") pod \"6cccf0cd-2042-4842-a2e6-471225fc7aac\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") "
Sep 18 14:23:40.857900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7hfl\" (UniqueName: \"kubernetes.io/projected/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-kube-api-access-k7hfl\") pod \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") "
Sep 18 14:23:40.857900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857766    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-home\") pod \"6cccf0cd-2042-4842-a2e6-471225fc7aac\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") "
Sep 18 14:23:40.857900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857802    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-home\") pod \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") "
Sep 18 14:23:40.857900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857829    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-home\") pod \"d1311a7e-85fa-48f1-b159-12119da8827d\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") "
Sep 18 14:23:40.857900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857856    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-model-cache\") pod \"d1311a7e-85fa-48f1-b159-12119da8827d\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") "
Sep 18 14:23:40.857900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857893    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cccf0cd-2042-4842-a2e6-471225fc7aac" (UID: "6cccf0cd-2042-4842-a2e6-471225fc7aac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.858519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.857908    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-tmp-dir\") pod \"6cccf0cd-2042-4842-a2e6-471225fc7aac\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") "
Sep 18 14:23:40.858519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858018    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-model-cache\") pod \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") "
Sep 18 14:23:40.858519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-home" (OuterVolumeSpecName: "home") pod "6cccf0cd-2042-4842-a2e6-471225fc7aac" (UID: "6cccf0cd-2042-4842-a2e6-471225fc7aac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.858519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858075    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-dshm\") pod \"6cccf0cd-2042-4842-a2e6-471225fc7aac\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") "
Sep 18 14:23:40.858519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cccf0cd-2042-4842-a2e6-471225fc7aac" (UID: "6cccf0cd-2042-4842-a2e6-471225fc7aac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.858519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858165    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-tmp-dir\") pod \"d1311a7e-85fa-48f1-b159-12119da8827d\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") "
Sep 18 14:23:40.858519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cccf0cd-2042-4842-a2e6-471225fc7aac-tls-certs\") pod \"6cccf0cd-2042-4842-a2e6-471225fc7aac\" (UID: \"6cccf0cd-2042-4842-a2e6-471225fc7aac\") "
Sep 18 14:23:40.858519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858231    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1311a7e-85fa-48f1-b159-12119da8827d-tls-certs\") pod \"d1311a7e-85fa-48f1-b159-12119da8827d\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") "
Sep 18 14:23:40.858519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858269    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tmp-dir\") pod \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\" (UID: \"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb\") "
Sep 18 14:23:40.858519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858305    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2ndg\" (UniqueName: \"kubernetes.io/projected/d1311a7e-85fa-48f1-b159-12119da8827d-kube-api-access-h2ndg\") pod \"d1311a7e-85fa-48f1-b159-12119da8827d\" (UID: \"d1311a7e-85fa-48f1-b159-12119da8827d\") "
Sep 18 14:23:40.858519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858369    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-home" (OuterVolumeSpecName: "home") pod "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" (UID: "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.859228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858547    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-home" (OuterVolumeSpecName: "home") pod "d1311a7e-85fa-48f1-b159-12119da8827d" (UID: "d1311a7e-85fa-48f1-b159-12119da8827d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.859228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858719    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1311a7e-85fa-48f1-b159-12119da8827d" (UID: "d1311a7e-85fa-48f1-b159-12119da8827d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.859228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858730    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.859228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858751    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.859228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858764    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.859228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858778    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.859228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.858792    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.859228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.859018    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" (UID: "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.860421 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.860380    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" (UID: "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:40.860421 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.860392    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-dshm" (OuterVolumeSpecName: "dshm") pod "d1311a7e-85fa-48f1-b159-12119da8827d" (UID: "d1311a7e-85fa-48f1-b159-12119da8827d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.860618 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.860584    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-dshm" (OuterVolumeSpecName: "dshm") pod "6cccf0cd-2042-4842-a2e6-471225fc7aac" (UID: "6cccf0cd-2042-4842-a2e6-471225fc7aac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.860680 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.860610    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" (UID: "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.860680 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.860612    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-kube-api-access-k7hfl" (OuterVolumeSpecName: "kube-api-access-k7hfl") pod "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" (UID: "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb"). InnerVolumeSpecName "kube-api-access-k7hfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:23:40.860786 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.860675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1311a7e-85fa-48f1-b159-12119da8827d" (UID: "d1311a7e-85fa-48f1-b159-12119da8827d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.860900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.860877    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-dshm" (OuterVolumeSpecName: "dshm") pod "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" (UID: "4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:23:40.861011 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.860984    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1311a7e-85fa-48f1-b159-12119da8827d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1311a7e-85fa-48f1-b159-12119da8827d" (UID: "d1311a7e-85fa-48f1-b159-12119da8827d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:40.861585 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.861563    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1311a7e-85fa-48f1-b159-12119da8827d-kube-api-access-h2ndg" (OuterVolumeSpecName: "kube-api-access-h2ndg") pod "d1311a7e-85fa-48f1-b159-12119da8827d" (UID: "d1311a7e-85fa-48f1-b159-12119da8827d"). InnerVolumeSpecName "kube-api-access-h2ndg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:23:40.861666 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.861585    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cccf0cd-2042-4842-a2e6-471225fc7aac-kube-api-access-f2dhf" (OuterVolumeSpecName: "kube-api-access-f2dhf") pod "6cccf0cd-2042-4842-a2e6-471225fc7aac" (UID: "6cccf0cd-2042-4842-a2e6-471225fc7aac"). InnerVolumeSpecName "kube-api-access-f2dhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:23:40.862292 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.862274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cccf0cd-2042-4842-a2e6-471225fc7aac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cccf0cd-2042-4842-a2e6-471225fc7aac" (UID: "6cccf0cd-2042-4842-a2e6-471225fc7aac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:23:40.959723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959632    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1311a7e-85fa-48f1-b159-12119da8827d-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.959723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959665    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.959723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959674    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2ndg\" (UniqueName: \"kubernetes.io/projected/d1311a7e-85fa-48f1-b159-12119da8827d-kube-api-access-h2ndg\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.959723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959681    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.959723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959690    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.959723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959698    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.959723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959706    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2dhf\" (UniqueName: \"kubernetes.io/projected/6cccf0cd-2042-4842-a2e6-471225fc7aac-kube-api-access-f2dhf\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.959723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959714    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7hfl\" (UniqueName: \"kubernetes.io/projected/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-kube-api-access-k7hfl\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.959723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959722    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.959723 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959730    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.960184 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959738    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cccf0cd-2042-4842-a2e6-471225fc7aac-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.960184 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959745    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1311a7e-85fa-48f1-b159-12119da8827d-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:40.960184 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:40.959753    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cccf0cd-2042-4842-a2e6-471225fc7aac-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:23:41.424465 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.424433    2563 generic.go:356] "Generic (PLEG): container finished" podID="4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" containerID="e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888" exitCode=0
Sep 18 14:23:41.424897 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.424492    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"
Sep 18 14:23:41.424897 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.424529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks" event={"ID":"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb","Type":"ContainerDied","Data":"e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888"}
Sep 18 14:23:41.424897 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.424586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks" event={"ID":"4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb","Type":"ContainerDied","Data":"e25d7fcf33ad6d1a9fe0d48cfcfa367da433af1a4136586fc72f9a0fa95b7b1e"}
Sep 18 14:23:41.424897 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.424612    2563 scope.go:117] "RemoveContainer" containerID="e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888"
Sep 18 14:23:41.426076 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.426055    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cccf0cd-2042-4842-a2e6-471225fc7aac" containerID="87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6" exitCode=0
Sep 18 14:23:41.426223 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.426102    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"
Sep 18 14:23:41.426223 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.426163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb" event={"ID":"6cccf0cd-2042-4842-a2e6-471225fc7aac","Type":"ContainerDied","Data":"87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6"}
Sep 18 14:23:41.426223 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.426198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb" event={"ID":"6cccf0cd-2042-4842-a2e6-471225fc7aac","Type":"ContainerDied","Data":"d533dcedffd4e57b74222af6653cf811342e50f559ac206df2b2af740cdd481e"}
Sep 18 14:23:41.427944 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.427840    2563 generic.go:356] "Generic (PLEG): container finished" podID="d1311a7e-85fa-48f1-b159-12119da8827d" containerID="61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288" exitCode=0
Sep 18 14:23:41.427944 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.427892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb" event={"ID":"d1311a7e-85fa-48f1-b159-12119da8827d","Type":"ContainerDied","Data":"61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288"}
Sep 18 14:23:41.427944 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.427900    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"
Sep 18 14:23:41.427944 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.427920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb" event={"ID":"d1311a7e-85fa-48f1-b159-12119da8827d","Type":"ContainerDied","Data":"c827f8286f1eb2d0621162e445a9aa5418a8be15735747b336db6c0c83b13adb"}
Sep 18 14:23:41.436781 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.436666    2563 scope.go:117] "RemoveContainer" containerID="e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888"
Sep 18 14:23:41.437052 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:23:41.437029    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888\": container with ID starting with e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888 not found: ID does not exist" containerID="e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888"
Sep 18 14:23:41.437168 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.437060    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888"} err="failed to get container status \"e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888\": rpc error: code = NotFound desc = could not find container \"e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888\": container with ID starting with e00d048b124cddf0dadded083369ab80c2590193324e79c902d90587cebfd888 not found: ID does not exist"
Sep 18 14:23:41.437168 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.437077    2563 scope.go:117] "RemoveContainer" containerID="87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6"
Sep 18 14:23:41.445714 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.445697    2563 scope.go:117] "RemoveContainer" containerID="87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6"
Sep 18 14:23:41.445971 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:23:41.445952    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6\": container with ID starting with 87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6 not found: ID does not exist" containerID="87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6"
Sep 18 14:23:41.446009 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.445979    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6"} err="failed to get container status \"87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6\": rpc error: code = NotFound desc = could not find container \"87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6\": container with ID starting with 87c99c9870e694c7e473073270ef522e92816e9e8f820d2dd4336a186fc64db6 not found: ID does not exist"
Sep 18 14:23:41.446009 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.445994    2563 scope.go:117] "RemoveContainer" containerID="61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288"
Sep 18 14:23:41.450628 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.450603    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"]
Sep 18 14:23:41.456842 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.455420    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-w4-93827d-kserve-59d8464bcd-d9zks"]
Sep 18 14:23:41.458336 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.458319    2563 scope.go:117] "RemoveContainer" containerID="61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288"
Sep 18 14:23:41.458670 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:23:41.458652    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288\": container with ID starting with 61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288 not found: ID does not exist" containerID="61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288"
Sep 18 14:23:41.458749 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.458678    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288"} err="failed to get container status \"61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288\": rpc error: code = NotFound desc = could not find container \"61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288\": container with ID starting with 61fe2ef77da57f96f55c731ccd405e8507b10360bc7d5569d31b13b582eaa288 not found: ID does not exist"
Sep 18 14:23:41.468598 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.468568    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"]
Sep 18 14:23:41.471841 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.471814    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/simulator-w4-93827d-kserve-567bd967bb-qrlnb"]
Sep 18 14:23:41.483058 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.483029    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"]
Sep 18 14:23:41.485473 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:41.485447    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-93827d/distinct-2-w4-93827d-kserve-68f95d7bbf-lbtlb"]
Sep 18 14:23:42.289199 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:42.289165    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" path="/var/lib/kubelet/pods/4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb/volumes"
Sep 18 14:23:42.289605 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:42.289592    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cccf0cd-2042-4842-a2e6-471225fc7aac" path="/var/lib/kubelet/pods/6cccf0cd-2042-4842-a2e6-471225fc7aac/volumes"
Sep 18 14:23:42.289947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:42.289935    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1311a7e-85fa-48f1-b159-12119da8827d" path="/var/lib/kubelet/pods/d1311a7e-85fa-48f1-b159-12119da8827d/volumes"
Sep 18 14:23:42.401056 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:23:42.401022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:24:00.561779 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:00.561740    2563 generic.go:356] "Generic (PLEG): container finished" podID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerID="64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a" exitCode=0
Sep 18 14:24:00.561779 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:00.561782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" event={"ID":"3078f077-d672-44f4-89a0-97c1c5f4f2de","Type":"ContainerDied","Data":"64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a"}
Sep 18 14:24:00.562305 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:00.562216    2563 scope.go:117] "RemoveContainer" containerID="64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a"
Sep 18 14:24:00.563391 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:00.563358    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:24:00.563496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:00.563449    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:24:00.563560 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:00.563503    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:24:01.356827 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:01.356796    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:24:01.569203 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:01.569161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" event={"ID":"3078f077-d672-44f4-89a0-97c1c5f4f2de","Type":"ContainerStarted","Data":"d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8"}
Sep 18 14:24:01.828581 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:01.828545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:24:01.830023 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:01.829994    2563 patch_prober.go:28] interesting pod/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 18 14:24:01.830203 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:01.830070    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 18 14:24:02.823806 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:02.823757    2563 patch_prober.go:28] interesting pod/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 18 14:24:02.824359 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:02.823837    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 18 14:24:03.620670 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:03.620618    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:24:03.824931 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:03.824888    2563 patch_prober.go:28] interesting pod/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 18 14:24:03.825555 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:03.825522    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 18 14:24:04.428868 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.428820    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"]
Sep 18 14:24:04.429737 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.429700    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l" podUID="f2bfae85-d825-4107-a53b-c1e0084a29ce" containerName="maas-api" containerID="cri-o://ce1feade91ca13078422f3f22d29c4d17c887d5431d7ef35e5daa12390eaf80f" gracePeriod=30
Sep 18 14:24:04.591544 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.591506    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2bfae85-d825-4107-a53b-c1e0084a29ce" containerID="ce1feade91ca13078422f3f22d29c4d17c887d5431d7ef35e5daa12390eaf80f" exitCode=0
Sep 18 14:24:04.591834 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.591578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l" event={"ID":"f2bfae85-d825-4107-a53b-c1e0084a29ce","Type":"ContainerDied","Data":"ce1feade91ca13078422f3f22d29c4d17c887d5431d7ef35e5daa12390eaf80f"}
Sep 18 14:24:04.700234 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.700207    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:24:04.789073 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.789032    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-maas-api-tls\") pod \"f2bfae85-d825-4107-a53b-c1e0084a29ce\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") "
Sep 18 14:24:04.789298 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.789080    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bfae85-d825-4107-a53b-c1e0084a29ce-openshift-service-ca\") pod \"f2bfae85-d825-4107-a53b-c1e0084a29ce\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") "
Sep 18 14:24:04.789298 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.789116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-metrics-certs\") pod \"f2bfae85-d825-4107-a53b-c1e0084a29ce\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") "
Sep 18 14:24:04.789298 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.789228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwg28\" (UniqueName: \"kubernetes.io/projected/f2bfae85-d825-4107-a53b-c1e0084a29ce-kube-api-access-jwg28\") pod \"f2bfae85-d825-4107-a53b-c1e0084a29ce\" (UID: \"f2bfae85-d825-4107-a53b-c1e0084a29ce\") "
Sep 18 14:24:04.789549 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.789517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2bfae85-d825-4107-a53b-c1e0084a29ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2bfae85-d825-4107-a53b-c1e0084a29ce" (UID: "f2bfae85-d825-4107-a53b-c1e0084a29ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:24:04.791674 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.791635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f2bfae85-d825-4107-a53b-c1e0084a29ce" (UID: "f2bfae85-d825-4107-a53b-c1e0084a29ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:24:04.791674 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.791652    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bfae85-d825-4107-a53b-c1e0084a29ce-kube-api-access-jwg28" (OuterVolumeSpecName: "kube-api-access-jwg28") pod "f2bfae85-d825-4107-a53b-c1e0084a29ce" (UID: "f2bfae85-d825-4107-a53b-c1e0084a29ce"). InnerVolumeSpecName "kube-api-access-jwg28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:04.791674 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.791670    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2bfae85-d825-4107-a53b-c1e0084a29ce" (UID: "f2bfae85-d825-4107-a53b-c1e0084a29ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:24:04.825154 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.823354    2563 patch_prober.go:28] interesting pod/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 18 14:24:04.825154 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.823436    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 18 14:24:04.890057 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.890017    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2bfae85-d825-4107-a53b-c1e0084a29ce-openshift-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:04.890057 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.890050    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-metrics-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:04.890057 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.890062    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwg28\" (UniqueName: \"kubernetes.io/projected/f2bfae85-d825-4107-a53b-c1e0084a29ce-kube-api-access-jwg28\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:04.890326 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:04.890070    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2bfae85-d825-4107-a53b-c1e0084a29ce-maas-api-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:05.599700 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:05.599670    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"
Sep 18 14:24:05.599891 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:05.599669    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l" event={"ID":"f2bfae85-d825-4107-a53b-c1e0084a29ce","Type":"ContainerDied","Data":"3e30a18bcc154bd561e62e57bfda2bd3e3fd2cfb9fa15955d7edf4a413ba0afd"}
Sep 18 14:24:05.599891 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:05.599803    2563 scope.go:117] "RemoveContainer" containerID="ce1feade91ca13078422f3f22d29c4d17c887d5431d7ef35e5daa12390eaf80f"
Sep 18 14:24:05.636320 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:05.636280    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"]
Sep 18 14:24:05.640219 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:05.640181    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e01050-64d7cd8c9c-pms7l"]
Sep 18 14:24:05.823499 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:05.823454    2563 patch_prober.go:28] interesting pod/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Sep 18 14:24:05.823721 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:05.823524    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Sep 18 14:24:06.288783 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:06.288749    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2bfae85-d825-4107-a53b-c1e0084a29ce" path="/var/lib/kubelet/pods/f2bfae85-d825-4107-a53b-c1e0084a29ce/volumes"
Sep 18 14:24:06.827473 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:06.827435    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:24:06.827918 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:06.827897    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:24:06.828700 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:06.828681    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:24:22.410538 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:22.410503    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:24:26.951300 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:26.951266    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"]
Sep 18 14:24:26.951779 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:26.951547    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9" podUID="ffac570e-c126-450e-8da0-9a9e766ebb0f" containerName="maas-api" containerID="cri-o://4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21" gracePeriod=30
Sep 18 14:24:27.200504 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.200477    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:24:27.312148 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.312036    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-maas-api-tls\") pod \"ffac570e-c126-450e-8da0-9a9e766ebb0f\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") "
Sep 18 14:24:27.312148 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.312085    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l9t8\" (UniqueName: \"kubernetes.io/projected/ffac570e-c126-450e-8da0-9a9e766ebb0f-kube-api-access-5l9t8\") pod \"ffac570e-c126-450e-8da0-9a9e766ebb0f\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") "
Sep 18 14:24:27.312384 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.312173    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-metrics-certs\") pod \"ffac570e-c126-450e-8da0-9a9e766ebb0f\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") "
Sep 18 14:24:27.312384 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.312228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffac570e-c126-450e-8da0-9a9e766ebb0f-openshift-service-ca\") pod \"ffac570e-c126-450e-8da0-9a9e766ebb0f\" (UID: \"ffac570e-c126-450e-8da0-9a9e766ebb0f\") "
Sep 18 14:24:27.312654 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.312624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffac570e-c126-450e-8da0-9a9e766ebb0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ffac570e-c126-450e-8da0-9a9e766ebb0f" (UID: "ffac570e-c126-450e-8da0-9a9e766ebb0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:24:27.314353 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.314325    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ffac570e-c126-450e-8da0-9a9e766ebb0f" (UID: "ffac570e-c126-450e-8da0-9a9e766ebb0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:24:27.314482 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.314446    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ffac570e-c126-450e-8da0-9a9e766ebb0f" (UID: "ffac570e-c126-450e-8da0-9a9e766ebb0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:24:27.314482 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.314448    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffac570e-c126-450e-8da0-9a9e766ebb0f-kube-api-access-5l9t8" (OuterVolumeSpecName: "kube-api-access-5l9t8") pod "ffac570e-c126-450e-8da0-9a9e766ebb0f" (UID: "ffac570e-c126-450e-8da0-9a9e766ebb0f"). InnerVolumeSpecName "kube-api-access-5l9t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:27.414123 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.414011    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffac570e-c126-450e-8da0-9a9e766ebb0f-openshift-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:27.414123 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.414045    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-maas-api-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:27.414123 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.414060    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l9t8\" (UniqueName: \"kubernetes.io/projected/ffac570e-c126-450e-8da0-9a9e766ebb0f-kube-api-access-5l9t8\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:27.414123 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.414074    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffac570e-c126-450e-8da0-9a9e766ebb0f-metrics-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:27.734419 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.734380    2563 generic.go:356] "Generic (PLEG): container finished" podID="ffac570e-c126-450e-8da0-9a9e766ebb0f" containerID="4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21" exitCode=0
Sep 18 14:24:27.734626 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.734452    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"
Sep 18 14:24:27.734626 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.734455    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9" event={"ID":"ffac570e-c126-450e-8da0-9a9e766ebb0f","Type":"ContainerDied","Data":"4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21"}
Sep 18 14:24:27.734626 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.734575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9" event={"ID":"ffac570e-c126-450e-8da0-9a9e766ebb0f","Type":"ContainerDied","Data":"5be9db67bfd621c9e3f0be71e142030b4ae69282c70ba912e78a29cd637d1869"}
Sep 18 14:24:27.734626 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.734603    2563 scope.go:117] "RemoveContainer" containerID="4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21"
Sep 18 14:24:27.744059 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.744041    2563 scope.go:117] "RemoveContainer" containerID="4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21"
Sep 18 14:24:27.744382 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:24:27.744363    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21\": container with ID starting with 4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21 not found: ID does not exist" containerID="4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21"
Sep 18 14:24:27.744463 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.744389    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21"} err="failed to get container status \"4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21\": rpc error: code = NotFound desc = could not find container \"4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21\": container with ID starting with 4a1bcbb1cf1e0d33ccc6f0e1b61a6aced301045d34957fb2662c3cb027d36f21 not found: ID does not exist"
Sep 18 14:24:27.764565 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.764530    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"]
Sep 18 14:24:27.768853 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:27.768827    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e9388b-59f554658b-5rnw9"]
Sep 18 14:24:28.288445 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:28.288414    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffac570e-c126-450e-8da0-9a9e766ebb0f" path="/var/lib/kubelet/pods/ffac570e-c126-450e-8da0-9a9e766ebb0f/volumes"
Sep 18 14:24:34.569681 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:34.569642    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"]
Sep 18 14:24:34.570105 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:34.569856    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy" containerID="cri-o://d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8" gracePeriod=30
Sep 18 14:24:36.625232 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:36.625194    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:24:36.830347 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:36.830310    2563 patch_prober.go:28] interesting pod/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 18 14:24:36.830512 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:36.830384    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 14:24:39.817448 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.817419    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:24:39.818760 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.818717    2563 generic.go:356] "Generic (PLEG): container finished" podID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerID="d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8" exitCode=0
Sep 18 14:24:39.818879 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.818788    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" event={"ID":"3078f077-d672-44f4-89a0-97c1c5f4f2de","Type":"ContainerDied","Data":"d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8"}
Sep 18 14:24:39.818879 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.818848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw" event={"ID":"3078f077-d672-44f4-89a0-97c1c5f4f2de","Type":"ContainerDied","Data":"62e460057ea76a4432775558ba6a9ef6a9130a763872301e90bfe3b3c3434ca6"}
Sep 18 14:24:39.818879 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.818871    2563 scope.go:117] "RemoveContainer" containerID="d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8"
Sep 18 14:24:39.828142 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.828108    2563 scope.go:117] "RemoveContainer" containerID="64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a"
Sep 18 14:24:39.832500 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832441    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-socket\") pod \"3078f077-d672-44f4-89a0-97c1c5f4f2de\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") "
Sep 18 14:24:39.832500 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-certs\") pod \"3078f077-d672-44f4-89a0-97c1c5f4f2de\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") "
Sep 18 14:24:39.832669 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832548    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-token\") pod \"3078f077-d672-44f4-89a0-97c1c5f4f2de\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") "
Sep 18 14:24:39.832669 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-credential-socket\") pod \"3078f077-d672-44f4-89a0-97c1c5f4f2de\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") "
Sep 18 14:24:39.832669 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832626    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-data\") pod \"3078f077-d672-44f4-89a0-97c1c5f4f2de\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") "
Sep 18 14:24:39.832822 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832694    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3078f077-d672-44f4-89a0-97c1c5f4f2de-istiod-ca-cert\") pod \"3078f077-d672-44f4-89a0-97c1c5f4f2de\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") "
Sep 18 14:24:39.832822 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3078f077-d672-44f4-89a0-97c1c5f4f2de" (UID: "3078f077-d672-44f4-89a0-97c1c5f4f2de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:39.832822 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832720    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtgwn\" (UniqueName: \"kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-kube-api-access-rtgwn\") pod \"3078f077-d672-44f4-89a0-97c1c5f4f2de\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") "
Sep 18 14:24:39.832822 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832755    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-podinfo\") pod \"3078f077-d672-44f4-89a0-97c1c5f4f2de\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") "
Sep 18 14:24:39.832822 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832806    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-envoy\") pod \"3078f077-d672-44f4-89a0-97c1c5f4f2de\" (UID: \"3078f077-d672-44f4-89a0-97c1c5f4f2de\") "
Sep 18 14:24:39.833061 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832861    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3078f077-d672-44f4-89a0-97c1c5f4f2de" (UID: "3078f077-d672-44f4-89a0-97c1c5f4f2de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:39.833061 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832914    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-data" (OuterVolumeSpecName: "istio-data") pod "3078f077-d672-44f4-89a0-97c1c5f4f2de" (UID: "3078f077-d672-44f4-89a0-97c1c5f4f2de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:39.833061 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.832931    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3078f077-d672-44f4-89a0-97c1c5f4f2de" (UID: "3078f077-d672-44f4-89a0-97c1c5f4f2de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:39.833061 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.833034    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3078f077-d672-44f4-89a0-97c1c5f4f2de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3078f077-d672-44f4-89a0-97c1c5f4f2de" (UID: "3078f077-d672-44f4-89a0-97c1c5f4f2de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:24:39.833320 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.833190    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3078f077-d672-44f4-89a0-97c1c5f4f2de-istiod-ca-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:39.833320 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.833213    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:39.833320 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.833225    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-workload-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:39.833320 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.833237    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-credential-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:39.833320 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.833250    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-data\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:39.835067 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.835044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-token" (OuterVolumeSpecName: "istio-token") pod "3078f077-d672-44f4-89a0-97c1c5f4f2de" (UID: "3078f077-d672-44f4-89a0-97c1c5f4f2de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:39.835207 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.835189    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3078f077-d672-44f4-89a0-97c1c5f4f2de" (UID: "3078f077-d672-44f4-89a0-97c1c5f4f2de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:24:39.835254 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.835226    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3078f077-d672-44f4-89a0-97c1c5f4f2de" (UID: "3078f077-d672-44f4-89a0-97c1c5f4f2de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:24:39.835389 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.835370    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-kube-api-access-rtgwn" (OuterVolumeSpecName: "kube-api-access-rtgwn") pod "3078f077-d672-44f4-89a0-97c1c5f4f2de" (UID: "3078f077-d672-44f4-89a0-97c1c5f4f2de"). InnerVolumeSpecName "kube-api-access-rtgwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:39.838355 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.838336    2563 scope.go:117] "RemoveContainer" containerID="d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8"
Sep 18 14:24:39.838707 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:24:39.838686    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8\": container with ID starting with d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8 not found: ID does not exist" containerID="d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8"
Sep 18 14:24:39.838776 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.838715    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8"} err="failed to get container status \"d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8\": rpc error: code = NotFound desc = could not find container \"d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8\": container with ID starting with d8a66848b5b9de064d3d551da658df7172764641b82d1210b1c5d7b9d2ca96a8 not found: ID does not exist"
Sep 18 14:24:39.838776 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.838740    2563 scope.go:117] "RemoveContainer" containerID="64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a"
Sep 18 14:24:39.839115 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:24:39.839079    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a\": container with ID starting with 64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a not found: ID does not exist" containerID="64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a"
Sep 18 14:24:39.839422 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.839111    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a"} err="failed to get container status \"64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a\": rpc error: code = NotFound desc = could not find container \"64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a\": container with ID starting with 64556a8bd532536cb7a449990e00c98d0579bc893ac88cb563b4b13a5af5497a not found: ID does not exist"
Sep 18 14:24:39.933901 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.933861    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-token\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:39.933901 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.933893    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtgwn\" (UniqueName: \"kubernetes.io/projected/3078f077-d672-44f4-89a0-97c1c5f4f2de-kube-api-access-rtgwn\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:39.933901 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.933904    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-podinfo\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:39.934177 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:39.933913    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3078f077-d672-44f4-89a0-97c1c5f4f2de-istio-envoy\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:40.826439 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:40.826407    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"
Sep 18 14:24:40.844846 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:40.844816    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"]
Sep 18 14:24:40.854464 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:40.854441    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e9388b-openshift-default-69655f4d64-8zwxw"]
Sep 18 14:24:42.288051 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:42.288015    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" path="/var/lib/kubelet/pods/3078f077-d672-44f4-89a0-97c1c5f4f2de/volumes"
Sep 18 14:24:45.384337 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.384303    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"]
Sep 18 14:24:45.384735 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.384542    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj" podUID="153fae7e-a7a2-4bd5-9c89-5a1b405b6b64" containerName="maas-api" containerID="cri-o://55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187" gracePeriod=30
Sep 18 14:24:45.632181 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.632118    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:24:45.684864 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.684834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-openshift-service-ca\") pod \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") "
Sep 18 14:24:45.685033 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.684880    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hkgw\" (UniqueName: \"kubernetes.io/projected/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-kube-api-access-6hkgw\") pod \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") "
Sep 18 14:24:45.685033 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.684952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls\") pod \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") "
Sep 18 14:24:45.685033 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.685029    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-metrics-certs\") pod \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\" (UID: \"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64\") "
Sep 18 14:24:45.685314 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.685286    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "153fae7e-a7a2-4bd5-9c89-5a1b405b6b64" (UID: "153fae7e-a7a2-4bd5-9c89-5a1b405b6b64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:24:45.687271 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.687246    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "153fae7e-a7a2-4bd5-9c89-5a1b405b6b64" (UID: "153fae7e-a7a2-4bd5-9c89-5a1b405b6b64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:24:45.687370 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.687320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "153fae7e-a7a2-4bd5-9c89-5a1b405b6b64" (UID: "153fae7e-a7a2-4bd5-9c89-5a1b405b6b64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:24:45.687416 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.687358    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-kube-api-access-6hkgw" (OuterVolumeSpecName: "kube-api-access-6hkgw") pod "153fae7e-a7a2-4bd5-9c89-5a1b405b6b64" (UID: "153fae7e-a7a2-4bd5-9c89-5a1b405b6b64"). InnerVolumeSpecName "kube-api-access-6hkgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:45.786446 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.786413    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-metrics-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:45.786446 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.786442    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-openshift-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:45.786446 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.786453    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hkgw\" (UniqueName: \"kubernetes.io/projected/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-kube-api-access-6hkgw\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:45.786663 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.786462    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64-maas-api-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:45.858531 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.858496    2563 generic.go:356] "Generic (PLEG): container finished" podID="153fae7e-a7a2-4bd5-9c89-5a1b405b6b64" containerID="55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187" exitCode=0
Sep 18 14:24:45.858699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.858543    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"
Sep 18 14:24:45.858699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.858586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj" event={"ID":"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64","Type":"ContainerDied","Data":"55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187"}
Sep 18 14:24:45.858699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.858643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj" event={"ID":"153fae7e-a7a2-4bd5-9c89-5a1b405b6b64","Type":"ContainerDied","Data":"3260d672fbaeb669917df1e4e9823a7f5313dd9d9321901471de26f19edc95c1"}
Sep 18 14:24:45.858699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.858674    2563 scope.go:117] "RemoveContainer" containerID="55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187"
Sep 18 14:24:45.868506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.868487    2563 scope.go:117] "RemoveContainer" containerID="55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187"
Sep 18 14:24:45.868811 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:24:45.868795    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187\": container with ID starting with 55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187 not found: ID does not exist" containerID="55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187"
Sep 18 14:24:45.868848 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.868820    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187"} err="failed to get container status \"55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187\": rpc error: code = NotFound desc = could not find container \"55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187\": container with ID starting with 55156aab2fd66b6ca491124227c13bed14dbc85725393cdca247c54f4a37f187 not found: ID does not exist"
Sep 18 14:24:45.901405 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.901372    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"]
Sep 18 14:24:45.914768 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:45.914737    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-526b97-575ffd4666-k9gkj"]
Sep 18 14:24:46.288388 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:46.288346    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="153fae7e-a7a2-4bd5-9c89-5a1b405b6b64" path="/var/lib/kubelet/pods/153fae7e-a7a2-4bd5-9c89-5a1b405b6b64/volumes"
Sep 18 14:24:49.926185 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:49.926151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:24:51.916209 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916170    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-686794"]
Sep 18 14:24:51.916712 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916692    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" containerName="main"
Sep 18 14:24:51.916796 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916716    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" containerName="main"
Sep 18 14:24:51.916796 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916739    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1311a7e-85fa-48f1-b159-12119da8827d" containerName="main"
Sep 18 14:24:51.916796 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916747    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1311a7e-85fa-48f1-b159-12119da8827d" containerName="main"
Sep 18 14:24:51.916796 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916759    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cccf0cd-2042-4842-a2e6-471225fc7aac" containerName="main"
Sep 18 14:24:51.916796 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916768    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cccf0cd-2042-4842-a2e6-471225fc7aac" containerName="main"
Sep 18 14:24:51.916796 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916779    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy"
Sep 18 14:24:51.916796 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916787    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916799    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2bfae85-d825-4107-a53b-c1e0084a29ce" containerName="maas-api"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916811    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2bfae85-d825-4107-a53b-c1e0084a29ce" containerName="maas-api"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916822    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffac570e-c126-450e-8da0-9a9e766ebb0f" containerName="maas-api"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916831    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffac570e-c126-450e-8da0-9a9e766ebb0f" containerName="maas-api"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916842    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916851    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916862    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="153fae7e-a7a2-4bd5-9c89-5a1b405b6b64" containerName="maas-api"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916871    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="153fae7e-a7a2-4bd5-9c89-5a1b405b6b64" containerName="maas-api"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916972    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2bfae85-d825-4107-a53b-c1e0084a29ce" containerName="maas-api"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916985    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1311a7e-85fa-48f1-b159-12119da8827d" containerName="main"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.916995    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="153fae7e-a7a2-4bd5-9c89-5a1b405b6b64" containerName="maas-api"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.917004    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ecbdab2-1c44-4d9d-9cb0-7e40f3f009fb" containerName="main"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.917013    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cccf0cd-2042-4842-a2e6-471225fc7aac" containerName="main"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.917022    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.917031    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffac570e-c126-450e-8da0-9a9e766ebb0f" containerName="maas-api"
Sep 18 14:24:51.917160 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.917041    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3078f077-d672-44f4-89a0-97c1c5f4f2de" containerName="istio-proxy"
Sep 18 14:24:51.921671 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.921651    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-686794"
Sep 18 14:24:51.924341 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.924318    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r4cmx\""
Sep 18 14:24:51.927117 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:51.927095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-686794"]
Sep 18 14:24:52.050378 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:52.050340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tptcj\" (UniqueName: \"kubernetes.io/projected/8bb64b52-e920-4c07-9573-e3b5912b338f-kube-api-access-tptcj\") pod \"test-curl-7761-686794\" (UID: \"8bb64b52-e920-4c07-9573-e3b5912b338f\") " pod="opendatahub/test-curl-7761-686794"
Sep 18 14:24:52.151102 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:52.151058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tptcj\" (UniqueName: \"kubernetes.io/projected/8bb64b52-e920-4c07-9573-e3b5912b338f-kube-api-access-tptcj\") pod \"test-curl-7761-686794\" (UID: \"8bb64b52-e920-4c07-9573-e3b5912b338f\") " pod="opendatahub/test-curl-7761-686794"
Sep 18 14:24:52.162839 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:52.162807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tptcj\" (UniqueName: \"kubernetes.io/projected/8bb64b52-e920-4c07-9573-e3b5912b338f-kube-api-access-tptcj\") pod \"test-curl-7761-686794\" (UID: \"8bb64b52-e920-4c07-9573-e3b5912b338f\") " pod="opendatahub/test-curl-7761-686794"
Sep 18 14:24:52.233113 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:52.233081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-686794"
Sep 18 14:24:52.504515 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:52.504479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-686794"]
Sep 18 14:24:52.505571 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:24:52.505540    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bb64b52_e920_4c07_9573_e3b5912b338f.slice/crio-6a0010109ce9e0bcb860c4ced1fff02ad6d67feafc91fc5aea837fc3d01c87dd WatchSource:0}: Error finding container 6a0010109ce9e0bcb860c4ced1fff02ad6d67feafc91fc5aea837fc3d01c87dd: Status 404 returned error can't find the container with id 6a0010109ce9e0bcb860c4ced1fff02ad6d67feafc91fc5aea837fc3d01c87dd
Sep 18 14:24:52.903505 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:52.903402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-686794" event={"ID":"8bb64b52-e920-4c07-9573-e3b5912b338f","Type":"ContainerStarted","Data":"8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c"}
Sep 18 14:24:52.903505 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:52.903454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-686794" event={"ID":"8bb64b52-e920-4c07-9573-e3b5912b338f","Type":"ContainerStarted","Data":"6a0010109ce9e0bcb860c4ced1fff02ad6d67feafc91fc5aea837fc3d01c87dd"}
Sep 18 14:24:52.917931 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:52.917858    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7761-686794" podStartSLOduration=1.917842466 podStartE2EDuration="1.917842466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:24:52.916888956 +0000 UTC m=+1581.073409234" watchObservedRunningTime="2026-09-18 14:24:52.917842466 +0000 UTC m=+1581.074362779"
Sep 18 14:24:53.531483 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:53.531445    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-686794"]
Sep 18 14:24:53.548222 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:53.548118    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-686794"]
Sep 18 14:24:53.909419 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:24:53.909337    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7761-686794" container="test-curl-7761-686794"
Sep 18 14:24:54.915195 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:54.915148    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7761-686794" podUID="8bb64b52-e920-4c07-9573-e3b5912b338f" containerName="test-curl-7761-686794" containerID="cri-o://8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c" gracePeriod=2
Sep 18 14:24:54.928157 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:54.928110    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"]
Sep 18 14:24:55.933566 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:55.933532    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"]
Sep 18 14:24:57.089413 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:57.089384    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-686794"
Sep 18 14:24:57.197478 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:57.197441    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tptcj\" (UniqueName: \"kubernetes.io/projected/8bb64b52-e920-4c07-9573-e3b5912b338f-kube-api-access-tptcj\") pod \"8bb64b52-e920-4c07-9573-e3b5912b338f\" (UID: \"8bb64b52-e920-4c07-9573-e3b5912b338f\") "
Sep 18 14:24:57.199531 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:57.199504    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb64b52-e920-4c07-9573-e3b5912b338f-kube-api-access-tptcj" (OuterVolumeSpecName: "kube-api-access-tptcj") pod "8bb64b52-e920-4c07-9573-e3b5912b338f" (UID: "8bb64b52-e920-4c07-9573-e3b5912b338f"). InnerVolumeSpecName "kube-api-access-tptcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:24:57.298752 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:57.298709    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tptcj\" (UniqueName: \"kubernetes.io/projected/8bb64b52-e920-4c07-9573-e3b5912b338f-kube-api-access-tptcj\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:24:57.934959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:57.934920    2563 generic.go:356] "Generic (PLEG): container finished" podID="8bb64b52-e920-4c07-9573-e3b5912b338f" containerID="8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c" exitCode=137
Sep 18 14:24:57.934959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:57.934962    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-686794"
Sep 18 14:24:57.935266 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:57.935019    2563 scope.go:117] "RemoveContainer" containerID="8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c"
Sep 18 14:24:57.944503 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:57.944485    2563 scope.go:117] "RemoveContainer" containerID="8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c"
Sep 18 14:24:57.944752 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:24:57.944732    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c\": container with ID starting with 8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c not found: ID does not exist" containerID="8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c"
Sep 18 14:24:57.944787 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:57.944761    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c"} err="failed to get container status \"8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c\": rpc error: code = NotFound desc = could not find container \"8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c\": container with ID starting with 8a1a5783c88921bb954a86337690413eb6c39897bd8c36b0c88e5be67370b09c not found: ID does not exist"
Sep 18 14:24:57.945874 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:24:57.945852    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7761-686794\" is forbidden: User \"system:node:ip-10-0-139-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-55.ec2.internal' and this object" podUID="8bb64b52-e920-4c07-9573-e3b5912b338f" pod="opendatahub/test-curl-7761-686794"
Sep 18 14:24:58.288294 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:24:58.288211    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bb64b52-e920-4c07-9573-e3b5912b338f" path="/var/lib/kubelet/pods/8bb64b52-e920-4c07-9573-e3b5912b338f/volumes"
Sep 18 14:25:06.018509 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:06.018475    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:25:09.946735 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:09.946682    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c" podUID="2b4eaf94-8355-4df0-9f78-cbcc43ec6142" containerName="main" containerID="cri-o://64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50" gracePeriod=45
Sep 18 14:25:10.191997 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.191970    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:25:10.315082 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.315006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-model-cache\") pod \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") "
Sep 18 14:25:10.315082 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.315037    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tmp-dir\") pod \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") "
Sep 18 14:25:10.315082 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.315066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-dshm\") pod \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") "
Sep 18 14:25:10.315393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.315106    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkmn6\" (UniqueName: \"kubernetes.io/projected/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-kube-api-access-wkmn6\") pod \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") "
Sep 18 14:25:10.315393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.315187    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tls-certs\") pod \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") "
Sep 18 14:25:10.315393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.315209    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-home\") pod \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\" (UID: \"2b4eaf94-8355-4df0-9f78-cbcc43ec6142\") "
Sep 18 14:25:10.315393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.315349    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b4eaf94-8355-4df0-9f78-cbcc43ec6142" (UID: "2b4eaf94-8355-4df0-9f78-cbcc43ec6142"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:10.315393 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.315363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b4eaf94-8355-4df0-9f78-cbcc43ec6142" (UID: "2b4eaf94-8355-4df0-9f78-cbcc43ec6142"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:10.315646 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.315523    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-home" (OuterVolumeSpecName: "home") pod "2b4eaf94-8355-4df0-9f78-cbcc43ec6142" (UID: "2b4eaf94-8355-4df0-9f78-cbcc43ec6142"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:10.315646 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.315593    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:10.315646 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.315614    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:10.317475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.317454    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-dshm" (OuterVolumeSpecName: "dshm") pod "2b4eaf94-8355-4df0-9f78-cbcc43ec6142" (UID: "2b4eaf94-8355-4df0-9f78-cbcc43ec6142"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:10.317585 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.317526    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b4eaf94-8355-4df0-9f78-cbcc43ec6142" (UID: "2b4eaf94-8355-4df0-9f78-cbcc43ec6142"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:25:10.317585 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.317542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-kube-api-access-wkmn6" (OuterVolumeSpecName: "kube-api-access-wkmn6") pod "2b4eaf94-8355-4df0-9f78-cbcc43ec6142" (UID: "2b4eaf94-8355-4df0-9f78-cbcc43ec6142"). InnerVolumeSpecName "kube-api-access-wkmn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:25:10.416340 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.416304    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:10.416340 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.416333    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkmn6\" (UniqueName: \"kubernetes.io/projected/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-kube-api-access-wkmn6\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:10.416340 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.416344    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:10.416665 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.416356    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b4eaf94-8355-4df0-9f78-cbcc43ec6142-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:10.952282 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:10.952242    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87" podUID="69235290-0a32-4d84-978e-30118c9285d9" containerName="main" containerID="cri-o://0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de" gracePeriod=45
Sep 18 14:25:11.012975 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.012947    2563 generic.go:356] "Generic (PLEG): container finished" podID="2b4eaf94-8355-4df0-9f78-cbcc43ec6142" containerID="64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50" exitCode=0
Sep 18 14:25:11.013092 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.012989    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"
Sep 18 14:25:11.013092 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.013007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c" event={"ID":"2b4eaf94-8355-4df0-9f78-cbcc43ec6142","Type":"ContainerDied","Data":"64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50"}
Sep 18 14:25:11.013092 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.013038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c" event={"ID":"2b4eaf94-8355-4df0-9f78-cbcc43ec6142","Type":"ContainerDied","Data":"012570afd0f8d64b9ca37364b79814cd054f948e62f748573abffc98d4cf72a7"}
Sep 18 14:25:11.013092 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.013055    2563 scope.go:117] "RemoveContainer" containerID="64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50"
Sep 18 14:25:11.038309 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.038285    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"]
Sep 18 14:25:11.042323 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.042302    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/auth-test-model-ae54af-kserve-859dff9959-bkf6c"]
Sep 18 14:25:11.063321 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.063300    2563 scope.go:117] "RemoveContainer" containerID="64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50"
Sep 18 14:25:11.063672 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:25:11.063649    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50\": container with ID starting with 64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50 not found: ID does not exist" containerID="64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50"
Sep 18 14:25:11.063769 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.063679    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50"} err="failed to get container status \"64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50\": rpc error: code = NotFound desc = could not find container \"64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50\": container with ID starting with 64264379cb53ddce94f4a136044e27924c0ecbcf09d977a9f4b55dd38d731e50 not found: ID does not exist"
Sep 18 14:25:11.207583 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.207515    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:25:11.323979 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.323948    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-model-cache\") pod \"69235290-0a32-4d84-978e-30118c9285d9\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") "
Sep 18 14:25:11.324170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.324012    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79l4r\" (UniqueName: \"kubernetes.io/projected/69235290-0a32-4d84-978e-30118c9285d9-kube-api-access-79l4r\") pod \"69235290-0a32-4d84-978e-30118c9285d9\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") "
Sep 18 14:25:11.324170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.324051    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69235290-0a32-4d84-978e-30118c9285d9-tls-certs\") pod \"69235290-0a32-4d84-978e-30118c9285d9\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") "
Sep 18 14:25:11.324170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.324065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-dshm\") pod \"69235290-0a32-4d84-978e-30118c9285d9\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") "
Sep 18 14:25:11.324170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.324106    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-home\") pod \"69235290-0a32-4d84-978e-30118c9285d9\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") "
Sep 18 14:25:11.324408 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.324180    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-tmp-dir\") pod \"69235290-0a32-4d84-978e-30118c9285d9\" (UID: \"69235290-0a32-4d84-978e-30118c9285d9\") "
Sep 18 14:25:11.324408 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.324355    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "69235290-0a32-4d84-978e-30118c9285d9" (UID: "69235290-0a32-4d84-978e-30118c9285d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:11.324554 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.324500    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:11.324624 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.324601    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-home" (OuterVolumeSpecName: "home") pod "69235290-0a32-4d84-978e-30118c9285d9" (UID: "69235290-0a32-4d84-978e-30118c9285d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:11.324690 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.324629    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69235290-0a32-4d84-978e-30118c9285d9" (UID: "69235290-0a32-4d84-978e-30118c9285d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:11.326299 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.326269    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69235290-0a32-4d84-978e-30118c9285d9-kube-api-access-79l4r" (OuterVolumeSpecName: "kube-api-access-79l4r") pod "69235290-0a32-4d84-978e-30118c9285d9" (UID: "69235290-0a32-4d84-978e-30118c9285d9"). InnerVolumeSpecName "kube-api-access-79l4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:25:11.326408 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.326305    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69235290-0a32-4d84-978e-30118c9285d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69235290-0a32-4d84-978e-30118c9285d9" (UID: "69235290-0a32-4d84-978e-30118c9285d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:25:11.326408 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.326366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-dshm" (OuterVolumeSpecName: "dshm") pod "69235290-0a32-4d84-978e-30118c9285d9" (UID: "69235290-0a32-4d84-978e-30118c9285d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:25:11.425562 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.425525    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79l4r\" (UniqueName: \"kubernetes.io/projected/69235290-0a32-4d84-978e-30118c9285d9-kube-api-access-79l4r\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:11.425562 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.425554    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69235290-0a32-4d84-978e-30118c9285d9-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:11.425562 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.425565    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:11.425562 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.425573    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:11.425820 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:11.425580    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69235290-0a32-4d84-978e-30118c9285d9-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:25:12.021099 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:12.021059    2563 generic.go:356] "Generic (PLEG): container finished" podID="69235290-0a32-4d84-978e-30118c9285d9" containerID="0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de" exitCode=0
Sep 18 14:25:12.021605 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:12.021106    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"
Sep 18 14:25:12.021605 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:12.021157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87" event={"ID":"69235290-0a32-4d84-978e-30118c9285d9","Type":"ContainerDied","Data":"0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de"}
Sep 18 14:25:12.021605 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:12.021211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87" event={"ID":"69235290-0a32-4d84-978e-30118c9285d9","Type":"ContainerDied","Data":"d91104c489c585151cb53e1178416c86fc27a5ec83d04ee347dbf61d685126de"}
Sep 18 14:25:12.021605 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:12.021231    2563 scope.go:117] "RemoveContainer" containerID="0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de"
Sep 18 14:25:12.031762 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:12.031741    2563 scope.go:117] "RemoveContainer" containerID="0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de"
Sep 18 14:25:12.032120 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:25:12.032098    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de\": container with ID starting with 0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de not found: ID does not exist" containerID="0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de"
Sep 18 14:25:12.032211 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:12.032143    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de"} err="failed to get container status \"0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de\": rpc error: code = NotFound desc = could not find container \"0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de\": container with ID starting with 0a173bbf50afb6cb08ac7f1d6373051bee97ffa940f7c639fb72bb69afe2b1de not found: ID does not exist"
Sep 18 14:25:12.043274 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:12.043238    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"]
Sep 18 14:25:12.048584 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:12.048555    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/auth-test-model-980661-kserve-86f7bc8dc5-b4b87"]
Sep 18 14:25:12.294207 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:12.290060    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b4eaf94-8355-4df0-9f78-cbcc43ec6142" path="/var/lib/kubelet/pods/2b4eaf94-8355-4df0-9f78-cbcc43ec6142/volumes"
Sep 18 14:25:12.294207 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:12.290724    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69235290-0a32-4d84-978e-30118c9285d9" path="/var/lib/kubelet/pods/69235290-0a32-4d84-978e-30118c9285d9/volumes"
Sep 18 14:25:18.536764 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.536719    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"]
Sep 18 14:25:18.537330 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.537278    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb64b52-e920-4c07-9573-e3b5912b338f" containerName="test-curl-7761-686794"
Sep 18 14:25:18.537330 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.537299    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb64b52-e920-4c07-9573-e3b5912b338f" containerName="test-curl-7761-686794"
Sep 18 14:25:18.537455 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.537334    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b4eaf94-8355-4df0-9f78-cbcc43ec6142" containerName="main"
Sep 18 14:25:18.537455 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.537342    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b4eaf94-8355-4df0-9f78-cbcc43ec6142" containerName="main"
Sep 18 14:25:18.537455 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.537359    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69235290-0a32-4d84-978e-30118c9285d9" containerName="main"
Sep 18 14:25:18.537455 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.537367    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69235290-0a32-4d84-978e-30118c9285d9" containerName="main"
Sep 18 14:25:18.537455 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.537454    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69235290-0a32-4d84-978e-30118c9285d9" containerName="main"
Sep 18 14:25:18.537687 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.537469    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b4eaf94-8355-4df0-9f78-cbcc43ec6142" containerName="main"
Sep 18 14:25:18.537687 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.537479    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb64b52-e920-4c07-9573-e3b5912b338f" containerName="test-curl-7761-686794"
Sep 18 14:25:18.542186 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.542163    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.545617 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.545584    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ae54af\"/\"sub-test-model-ae54af-kserve-self-signed-certs\""
Sep 18 14:25:18.545766 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.545621    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ae54af\"/\"kube-root-ca.crt\""
Sep 18 14:25:18.545766 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.545629    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ae54af\"/\"default-dockercfg-k6q9w\""
Sep 18 14:25:18.545766 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.545627    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ae54af\"/\"openshift-service-ca.crt\""
Sep 18 14:25:18.550234 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.550212    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"]
Sep 18 14:25:18.585622 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.585588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tmp-dir\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.585819 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.585639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tls-certs\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.585819 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.585735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d48kp\" (UniqueName: \"kubernetes.io/projected/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-kube-api-access-d48kp\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.585819 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.585787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-dshm\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.585945 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.585836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-home\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.585945 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.585857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-model-cache\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.687144 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.687100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d48kp\" (UniqueName: \"kubernetes.io/projected/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-kube-api-access-d48kp\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.687348 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.687172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-dshm\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.687348 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.687203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-home\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.687348 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.687219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-model-cache\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.687348 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.687269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tmp-dir\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.687348 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.687328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tls-certs\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.687687 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.687656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-home\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.687813 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.687687    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-model-cache\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.687813 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.687717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tmp-dir\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.689692 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.689671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-dshm\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.689974 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.689957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tls-certs\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.701151 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.697282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d48kp\" (UniqueName: \"kubernetes.io/projected/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-kube-api-access-d48kp\") pod \"sub-test-model-ae54af-kserve-66599888cf-vgd2z\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:18.852698 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:18.852589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:19.128699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:19.128669    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"]
Sep 18 14:25:19.129765 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:25:19.129740    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c1fefc4_2fc2_4ff4_800e_6ef998ecabcb.slice/crio-51602db05eec2e236be300bc81f0d6534c1a9421e19abe00943c36bf4d61a132 WatchSource:0}: Error finding container 51602db05eec2e236be300bc81f0d6534c1a9421e19abe00943c36bf4d61a132: Status 404 returned error can't find the container with id 51602db05eec2e236be300bc81f0d6534c1a9421e19abe00943c36bf4d61a132
Sep 18 14:25:20.068990 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:20.068952    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z" event={"ID":"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb","Type":"ContainerStarted","Data":"44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d"}
Sep 18 14:25:20.068990 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:20.068997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z" event={"ID":"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb","Type":"ContainerStarted","Data":"51602db05eec2e236be300bc81f0d6534c1a9421e19abe00943c36bf4d61a132"}
Sep 18 14:25:20.085902 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:20.085849    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z" podStartSLOduration=2.085833531 podStartE2EDuration="2.085833531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:25:20.085390339 +0000 UTC m=+1608.241910621" watchObservedRunningTime="2026-09-18 14:25:20.085833531 +0000 UTC m=+1608.242353810"
Sep 18 14:25:21.520922 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:21.520884    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:25:28.852740 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:28.852696    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:28.853388 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:28.852752    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:28.865642 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:28.865614    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:29.131416 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:29.131349    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:25:53.828284 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:25:53.828242    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:26:09.115940 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:09.115901    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:26:22.117566 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:22.117522    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:26:25.806947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:25.806911    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-2ed9e3"]
Sep 18 14:26:25.810470 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:25.810453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-2ed9e3"
Sep 18 14:26:25.813936 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:25.813911    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-r4cmx\""
Sep 18 14:26:25.824833 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:25.824807    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-2ed9e3"]
Sep 18 14:26:25.901689 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:25.901652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh2lk\" (UniqueName: \"kubernetes.io/projected/f280baf4-bf82-4ce0-8c9f-030320c1fdec-kube-api-access-lh2lk\") pod \"test-curl-7761-2ed9e3\" (UID: \"f280baf4-bf82-4ce0-8c9f-030320c1fdec\") " pod="opendatahub/test-curl-7761-2ed9e3"
Sep 18 14:26:26.002782 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:26.002742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh2lk\" (UniqueName: \"kubernetes.io/projected/f280baf4-bf82-4ce0-8c9f-030320c1fdec-kube-api-access-lh2lk\") pod \"test-curl-7761-2ed9e3\" (UID: \"f280baf4-bf82-4ce0-8c9f-030320c1fdec\") " pod="opendatahub/test-curl-7761-2ed9e3"
Sep 18 14:26:26.010534 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:26.010507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh2lk\" (UniqueName: \"kubernetes.io/projected/f280baf4-bf82-4ce0-8c9f-030320c1fdec-kube-api-access-lh2lk\") pod \"test-curl-7761-2ed9e3\" (UID: \"f280baf4-bf82-4ce0-8c9f-030320c1fdec\") " pod="opendatahub/test-curl-7761-2ed9e3"
Sep 18 14:26:26.119494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:26.119409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-2ed9e3"
Sep 18 14:26:26.390429 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:26.390400    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-2ed9e3"]
Sep 18 14:26:26.392430 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:26:26.392398    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf280baf4_bf82_4ce0_8c9f_030320c1fdec.slice/crio-e6962133ee1576a3465ec397dd54c122d85e8c0c7bf20dc9f2621c8a87eb89a6 WatchSource:0}: Error finding container e6962133ee1576a3465ec397dd54c122d85e8c0c7bf20dc9f2621c8a87eb89a6: Status 404 returned error can't find the container with id e6962133ee1576a3465ec397dd54c122d85e8c0c7bf20dc9f2621c8a87eb89a6
Sep 18 14:26:26.439773 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:26.439742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-2ed9e3" event={"ID":"f280baf4-bf82-4ce0-8c9f-030320c1fdec","Type":"ContainerStarted","Data":"e6962133ee1576a3465ec397dd54c122d85e8c0c7bf20dc9f2621c8a87eb89a6"}
Sep 18 14:26:27.452980 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:27.452924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-2ed9e3" event={"ID":"f280baf4-bf82-4ce0-8c9f-030320c1fdec","Type":"ContainerStarted","Data":"88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb"}
Sep 18 14:26:27.470822 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:27.470772    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7761-2ed9e3" podStartSLOduration=2.470755891 podStartE2EDuration="2.470755891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:26:27.468962437 +0000 UTC m=+1675.625482716" watchObservedRunningTime="2026-09-18 14:26:27.470755891 +0000 UTC m=+1675.627276170"
Sep 18 14:26:28.094292 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.094250    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-2ed9e3"]
Sep 18 14:26:28.100357 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.100322    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-2ed9e3"]
Sep 18 14:26:28.232971 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.232927    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-9a0e0d"]
Sep 18 14:26:28.233443 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.233420    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f280baf4-bf82-4ce0-8c9f-030320c1fdec" containerName="test-curl-7761-2ed9e3"
Sep 18 14:26:28.233551 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.233444    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f280baf4-bf82-4ce0-8c9f-030320c1fdec" containerName="test-curl-7761-2ed9e3"
Sep 18 14:26:28.233551 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.233536    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f280baf4-bf82-4ce0-8c9f-030320c1fdec" containerName="test-curl-7761-2ed9e3"
Sep 18 14:26:28.236718 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.236699    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9a0e0d"
Sep 18 14:26:28.247810 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.247781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-9a0e0d"]
Sep 18 14:26:28.326434 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.326396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtsqf\" (UniqueName: \"kubernetes.io/projected/d6f80c9a-7459-48bf-8879-a1702459d3ca-kube-api-access-vtsqf\") pod \"test-curl-7761-9a0e0d\" (UID: \"d6f80c9a-7459-48bf-8879-a1702459d3ca\") " pod="opendatahub/test-curl-7761-9a0e0d"
Sep 18 14:26:28.427607 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.427560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtsqf\" (UniqueName: \"kubernetes.io/projected/d6f80c9a-7459-48bf-8879-a1702459d3ca-kube-api-access-vtsqf\") pod \"test-curl-7761-9a0e0d\" (UID: \"d6f80c9a-7459-48bf-8879-a1702459d3ca\") " pod="opendatahub/test-curl-7761-9a0e0d"
Sep 18 14:26:28.436399 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.436368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtsqf\" (UniqueName: \"kubernetes.io/projected/d6f80c9a-7459-48bf-8879-a1702459d3ca-kube-api-access-vtsqf\") pod \"test-curl-7761-9a0e0d\" (UID: \"d6f80c9a-7459-48bf-8879-a1702459d3ca\") " pod="opendatahub/test-curl-7761-9a0e0d"
Sep 18 14:26:28.458469 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:26:28.458440    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7761-2ed9e3" container="test-curl-7761-2ed9e3"
Sep 18 14:26:28.546875 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.546774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9a0e0d"
Sep 18 14:26:28.831623 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:28.831592    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-9a0e0d"]
Sep 18 14:26:28.833347 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:26:28.833319    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6f80c9a_7459_48bf_8879_a1702459d3ca.slice/crio-2378c88593d77f2998ba6099ad43807925c74940cc36c1f73686b9c307260f52 WatchSource:0}: Error finding container 2378c88593d77f2998ba6099ad43807925c74940cc36c1f73686b9c307260f52: Status 404 returned error can't find the container with id 2378c88593d77f2998ba6099ad43807925c74940cc36c1f73686b9c307260f52
Sep 18 14:26:29.465381 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:29.465341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-9a0e0d" event={"ID":"d6f80c9a-7459-48bf-8879-a1702459d3ca","Type":"ContainerStarted","Data":"d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761"}
Sep 18 14:26:29.465773 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:29.465392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-9a0e0d" event={"ID":"d6f80c9a-7459-48bf-8879-a1702459d3ca","Type":"ContainerStarted","Data":"2378c88593d77f2998ba6099ad43807925c74940cc36c1f73686b9c307260f52"}
Sep 18 14:26:29.465773 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:29.465418    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7761-2ed9e3" podUID="f280baf4-bf82-4ce0-8c9f-030320c1fdec" containerName="test-curl-7761-2ed9e3" containerID="cri-o://88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb" gracePeriod=2
Sep 18 14:26:29.482245 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:29.482196    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7761-9a0e0d" podStartSLOduration=1.4821803789999999 podStartE2EDuration="1.482180379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:26:29.481098213 +0000 UTC m=+1677.637618516" watchObservedRunningTime="2026-09-18 14:26:29.482180379 +0000 UTC m=+1677.638700658"
Sep 18 14:26:30.084619 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:30.084575    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-9a0e0d"]
Sep 18 14:26:30.089660 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:30.089630    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-9a0e0d"]
Sep 18 14:26:30.470461 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:26:30.470432    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7761-9a0e0d" container="test-curl-7761-9a0e0d"
Sep 18 14:26:31.476788 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:31.476757    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7761-9a0e0d" podUID="d6f80c9a-7459-48bf-8879-a1702459d3ca" containerName="test-curl-7761-9a0e0d" containerID="cri-o://d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761" gracePeriod=2
Sep 18 14:26:31.641177 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:31.641150    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-2ed9e3"
Sep 18 14:26:31.761551 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:31.761479    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh2lk\" (UniqueName: \"kubernetes.io/projected/f280baf4-bf82-4ce0-8c9f-030320c1fdec-kube-api-access-lh2lk\") pod \"f280baf4-bf82-4ce0-8c9f-030320c1fdec\" (UID: \"f280baf4-bf82-4ce0-8c9f-030320c1fdec\") "
Sep 18 14:26:31.763587 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:31.763555    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f280baf4-bf82-4ce0-8c9f-030320c1fdec-kube-api-access-lh2lk" (OuterVolumeSpecName: "kube-api-access-lh2lk") pod "f280baf4-bf82-4ce0-8c9f-030320c1fdec" (UID: "f280baf4-bf82-4ce0-8c9f-030320c1fdec"). InnerVolumeSpecName "kube-api-access-lh2lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:26:31.862285 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:31.862254    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh2lk\" (UniqueName: \"kubernetes.io/projected/f280baf4-bf82-4ce0-8c9f-030320c1fdec-kube-api-access-lh2lk\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:26:32.064918 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:32.064844    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"]
Sep 18 14:26:32.288034 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:32.288000    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f280baf4-bf82-4ce0-8c9f-030320c1fdec" path="/var/lib/kubelet/pods/f280baf4-bf82-4ce0-8c9f-030320c1fdec/volumes"
Sep 18 14:26:32.483447 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:32.483414    2563 generic.go:356] "Generic (PLEG): container finished" podID="f280baf4-bf82-4ce0-8c9f-030320c1fdec" containerID="88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb" exitCode=137
Sep 18 14:26:32.483855 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:32.483455    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-2ed9e3"
Sep 18 14:26:32.483855 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:32.483461    2563 scope.go:117] "RemoveContainer" containerID="88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb"
Sep 18 14:26:32.492005 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:32.491987    2563 scope.go:117] "RemoveContainer" containerID="88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb"
Sep 18 14:26:32.492268 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:26:32.492248    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb\": container with ID starting with 88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb not found: ID does not exist" containerID="88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb"
Sep 18 14:26:32.492333 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:32.492279    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb"} err="failed to get container status \"88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb\": rpc error: code = NotFound desc = could not find container \"88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb\": container with ID starting with 88d7911a039684ef2dd1eb168316de73b79ae8c4221c04e075641ae5197428bb not found: ID does not exist"
Sep 18 14:26:33.659693 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:33.659667    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9a0e0d"
Sep 18 14:26:33.777262 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:33.777204    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtsqf\" (UniqueName: \"kubernetes.io/projected/d6f80c9a-7459-48bf-8879-a1702459d3ca-kube-api-access-vtsqf\") pod \"d6f80c9a-7459-48bf-8879-a1702459d3ca\" (UID: \"d6f80c9a-7459-48bf-8879-a1702459d3ca\") "
Sep 18 14:26:33.779089 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:33.779064    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f80c9a-7459-48bf-8879-a1702459d3ca-kube-api-access-vtsqf" (OuterVolumeSpecName: "kube-api-access-vtsqf") pod "d6f80c9a-7459-48bf-8879-a1702459d3ca" (UID: "d6f80c9a-7459-48bf-8879-a1702459d3ca"). InnerVolumeSpecName "kube-api-access-vtsqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:26:33.878676 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:33.878655    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtsqf\" (UniqueName: \"kubernetes.io/projected/d6f80c9a-7459-48bf-8879-a1702459d3ca-kube-api-access-vtsqf\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:26:34.288169 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:34.288108    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6f80c9a-7459-48bf-8879-a1702459d3ca" path="/var/lib/kubelet/pods/d6f80c9a-7459-48bf-8879-a1702459d3ca/volumes"
Sep 18 14:26:34.495923 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:34.495891    2563 generic.go:356] "Generic (PLEG): container finished" podID="d6f80c9a-7459-48bf-8879-a1702459d3ca" containerID="d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761" exitCode=137
Sep 18 14:26:34.496028 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:34.495937    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-9a0e0d"
Sep 18 14:26:34.496028 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:34.495945    2563 scope.go:117] "RemoveContainer" containerID="d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761"
Sep 18 14:26:34.504950 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:34.504932    2563 scope.go:117] "RemoveContainer" containerID="d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761"
Sep 18 14:26:34.505258 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:26:34.505230    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761\": container with ID starting with d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761 not found: ID does not exist" containerID="d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761"
Sep 18 14:26:34.505362 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:34.505262    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761"} err="failed to get container status \"d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761\": rpc error: code = NotFound desc = could not find container \"d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761\": container with ID starting with d924f534ebede5602f720d1d96380ca6ff0d67bee76a678603a35a35602ef761 not found: ID does not exist"
Sep 18 14:26:35.413943 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:35.413911    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:26:42.618998 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.618968    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"]
Sep 18 14:26:42.619603 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.619330    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f80c9a-7459-48bf-8879-a1702459d3ca" containerName="test-curl-7761-9a0e0d"
Sep 18 14:26:42.619603 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.619343    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f80c9a-7459-48bf-8879-a1702459d3ca" containerName="test-curl-7761-9a0e0d"
Sep 18 14:26:42.619603 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.619408    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f80c9a-7459-48bf-8879-a1702459d3ca" containerName="test-curl-7761-9a0e0d"
Sep 18 14:26:42.623643 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.623624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.626204 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.626178    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ae54af\"/\"rate-test-model-ae54af-kserve-self-signed-certs\""
Sep 18 14:26:42.630956 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.630930    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"]
Sep 18 14:26:42.745390 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.745369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tmp-dir\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.745496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.745396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-home\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.745496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.745416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tls-certs\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.745496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.745441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fqmh\" (UniqueName: \"kubernetes.io/projected/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-kube-api-access-6fqmh\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.745605 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.745518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-model-cache\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.745605 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.745554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-dshm\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.846799 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.846764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tmp-dir\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.846799 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.846803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-home\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.846973 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.846820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tls-certs\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.846973 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.846845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fqmh\" (UniqueName: \"kubernetes.io/projected/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-kube-api-access-6fqmh\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.846973 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.846872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-model-cache\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.846973 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.846903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-dshm\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.847228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.847206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-home\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.847280 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.847228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tmp-dir\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.847280 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.847271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-model-cache\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.849072 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.849051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-dshm\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.849496 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.849472    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tls-certs\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.854492 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.854477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fqmh\" (UniqueName: \"kubernetes.io/projected/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-kube-api-access-6fqmh\") pod \"rate-test-model-ae54af-kserve-67c65f8dc-5j7ks\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") " pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:42.934930 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:42.934901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:43.208290 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:43.208263    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"]
Sep 18 14:26:43.210201 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:26:43.210172    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda09d08e2_83e0_4ee6_9b14_65f00cb8c966.slice/crio-e09884c73829f97b9b57b0ad09f050c3b98b626962ea66174cb5f3e10c852623 WatchSource:0}: Error finding container e09884c73829f97b9b57b0ad09f050c3b98b626962ea66174cb5f3e10c852623: Status 404 returned error can't find the container with id e09884c73829f97b9b57b0ad09f050c3b98b626962ea66174cb5f3e10c852623
Sep 18 14:26:43.551918 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:43.551825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks" event={"ID":"a09d08e2-83e0-4ee6-9b14-65f00cb8c966","Type":"ContainerStarted","Data":"75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be"}
Sep 18 14:26:43.551918 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:43.551867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks" event={"ID":"a09d08e2-83e0-4ee6-9b14-65f00cb8c966","Type":"ContainerStarted","Data":"e09884c73829f97b9b57b0ad09f050c3b98b626962ea66174cb5f3e10c852623"}
Sep 18 14:26:43.570036 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:43.569978    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks" podStartSLOduration=1.569961942 podStartE2EDuration="1.569961942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:26:43.568323865 +0000 UTC m=+1691.724844168" watchObservedRunningTime="2026-09-18 14:26:43.569961942 +0000 UTC m=+1691.726482275"
Sep 18 14:26:47.083243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.083205    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z" podUID="7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" containerName="main" containerID="cri-o://44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d" gracePeriod=45
Sep 18 14:26:47.240083 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.240047    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:26:47.332781 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.332752    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:26:47.488430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.488395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d48kp\" (UniqueName: \"kubernetes.io/projected/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-kube-api-access-d48kp\") pod \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") "
Sep 18 14:26:47.488628 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.488457    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tls-certs\") pod \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") "
Sep 18 14:26:47.488628 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.488478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-home\") pod \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") "
Sep 18 14:26:47.488703 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.488662    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-model-cache\") pod \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") "
Sep 18 14:26:47.488737 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.488702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-dshm\") pod \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") "
Sep 18 14:26:47.488771 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.488747    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-home" (OuterVolumeSpecName: "home") pod "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" (UID: "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:26:47.488816 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.488778    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tmp-dir\") pod \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\" (UID: \"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb\") "
Sep 18 14:26:47.488931 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.488903    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" (UID: "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:26:47.489085 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.489051    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" (UID: "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:26:47.489246 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.489124    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:26:47.489246 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.489168    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:26:47.490713 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.490690    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-dshm" (OuterVolumeSpecName: "dshm") pod "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" (UID: "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:26:47.490812 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.490752    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-kube-api-access-d48kp" (OuterVolumeSpecName: "kube-api-access-d48kp") pod "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" (UID: "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb"). InnerVolumeSpecName "kube-api-access-d48kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:26:47.490812 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.490757    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" (UID: "7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:26:47.574220 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.574174    2563 generic.go:356] "Generic (PLEG): container finished" podID="7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" containerID="44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d" exitCode=0
Sep 18 14:26:47.574220 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.574223    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"
Sep 18 14:26:47.574428 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.574258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z" event={"ID":"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb","Type":"ContainerDied","Data":"44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d"}
Sep 18 14:26:47.574428 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.574301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z" event={"ID":"7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb","Type":"ContainerDied","Data":"51602db05eec2e236be300bc81f0d6534c1a9421e19abe00943c36bf4d61a132"}
Sep 18 14:26:47.574428 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.574319    2563 scope.go:117] "RemoveContainer" containerID="44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d"
Sep 18 14:26:47.583946 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.583924    2563 scope.go:117] "RemoveContainer" containerID="44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d"
Sep 18 14:26:47.584237 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:26:47.584219    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d\": container with ID starting with 44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d not found: ID does not exist" containerID="44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d"
Sep 18 14:26:47.584274 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.584248    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d"} err="failed to get container status \"44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d\": rpc error: code = NotFound desc = could not find container \"44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d\": container with ID starting with 44d4fbb6398fa4d0363b38cadfc8dd5f7acf5b7a75596e34f07c2117af67ef7d not found: ID does not exist"
Sep 18 14:26:47.590030 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.590012    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:26:47.590084 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.590032    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:26:47.590084 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.590041    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:26:47.590084 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.590049    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d48kp\" (UniqueName: \"kubernetes.io/projected/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb-kube-api-access-d48kp\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:26:47.595825 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.595803    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"]
Sep 18 14:26:47.599802 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:47.599780    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/sub-test-model-ae54af-kserve-66599888cf-vgd2z"]
Sep 18 14:26:48.287991 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:48.287957    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" path="/var/lib/kubelet/pods/7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb/volumes"
Sep 18 14:26:52.935250 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:52.935203    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:52.935250 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:52.935255    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:52.948256 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:52.948228    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:26:53.619034 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:26:53.619004    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:27:09.819338 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.819256    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"]
Sep 18 14:27:09.819820 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.819802    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" containerName="main"
Sep 18 14:27:09.819864 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.819823    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" containerName="main"
Sep 18 14:27:09.819938 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.819927    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1fefc4-2fc2-4ff4-800e-6ef998ecabcb" containerName="main"
Sep 18 14:27:09.823349 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.823330    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.826844 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.826819    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-980661\"/\"default-dockercfg-qvxhp\""
Sep 18 14:27:09.826971 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.826819    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-980661\"/\"kube-root-ca.crt\""
Sep 18 14:27:09.826971 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.826864    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-980661\"/\"openshift-service-ca.crt\""
Sep 18 14:27:09.826971 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.826864    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-980661\"/\"rate-test-model-980661-kserve-self-signed-certs\""
Sep 18 14:27:09.837179 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.837157    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"]
Sep 18 14:27:09.886475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.886443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfsch\" (UniqueName: \"kubernetes.io/projected/31c8f7fd-0f89-49e5-8bd8-1e988753b165-kube-api-access-mfsch\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.886583 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.886491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-dshm\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.886583 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.886526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-model-cache\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.886583 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.886551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tls-certs\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.886583 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.886569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-home\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.886780 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.886594    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tmp-dir\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.987307 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.987283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfsch\" (UniqueName: \"kubernetes.io/projected/31c8f7fd-0f89-49e5-8bd8-1e988753b165-kube-api-access-mfsch\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.987430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.987318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-dshm\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.987430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.987340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-model-cache\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.987430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.987359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tls-certs\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.987430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.987379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-home\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.987430 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.987400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tmp-dir\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.987772 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.987751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tmp-dir\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.987831 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.987775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-model-cache\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.987831 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.987815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-home\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.989563 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.989543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-dshm\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.989727 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.989712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tls-certs\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:09.995257 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:09.995235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfsch\" (UniqueName: \"kubernetes.io/projected/31c8f7fd-0f89-49e5-8bd8-1e988753b165-kube-api-access-mfsch\") pod \"rate-test-model-980661-kserve-6646f6b955-xds5t\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") " pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:10.134648 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:10.134595    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:10.405806 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:10.405780    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"]
Sep 18 14:27:10.407342 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:27:10.407316    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31c8f7fd_0f89_49e5_8bd8_1e988753b165.slice/crio-27662563e18e6da5f5dfd1b1f9158528c21a0d7040042039058d813622b81843 WatchSource:0}: Error finding container 27662563e18e6da5f5dfd1b1f9158528c21a0d7040042039058d813622b81843: Status 404 returned error can't find the container with id 27662563e18e6da5f5dfd1b1f9158528c21a0d7040042039058d813622b81843
Sep 18 14:27:10.710991 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:10.710954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t" event={"ID":"31c8f7fd-0f89-49e5-8bd8-1e988753b165","Type":"ContainerStarted","Data":"62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7"}
Sep 18 14:27:10.711123 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:10.711002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t" event={"ID":"31c8f7fd-0f89-49e5-8bd8-1e988753b165","Type":"ContainerStarted","Data":"27662563e18e6da5f5dfd1b1f9158528c21a0d7040042039058d813622b81843"}
Sep 18 14:27:10.728845 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:10.728790    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t" podStartSLOduration=1.728775088 podStartE2EDuration="1.728775088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:27:10.727090312 +0000 UTC m=+1718.883610591" watchObservedRunningTime="2026-09-18 14:27:10.728775088 +0000 UTC m=+1718.885295366"
Sep 18 14:27:13.422703 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:13.422663    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:27:20.135446 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:20.135394    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:20.135992 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:20.135659    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:20.148021 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:20.147996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:20.776099 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:20.776066    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:27:30.608936 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:30.608899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:27:41.312041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:41.311991    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:27:46.392701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:46.392667    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"]
Sep 18 14:27:47.329790 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:47.329749    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"]
Sep 18 14:27:53.010770 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.010733    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"]
Sep 18 14:27:53.015315 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.015287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.017738 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.017714    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3ab8a5-serving-cert\""
Sep 18 14:27:53.028824 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.028789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"]
Sep 18 14:27:53.067897 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.067871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/140d328c-1945-493d-b208-f89a0bd03b2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.067996 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.067905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.067996 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.067931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.068091 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.068048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlgf\" (UniqueName: \"kubernetes.io/projected/140d328c-1945-493d-b208-f89a0bd03b2d-kube-api-access-stlgf\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.168613 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.168588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/140d328c-1945-493d-b208-f89a0bd03b2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.168710 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.168621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.168710 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.168646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.168825 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.168707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlgf\" (UniqueName: \"kubernetes.io/projected/140d328c-1945-493d-b208-f89a0bd03b2d-kube-api-access-stlgf\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.169220 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.169189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/140d328c-1945-493d-b208-f89a0bd03b2d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.170928 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.170907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-metrics-certs\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.171021 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.170992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.176257 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.176235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlgf\" (UniqueName: \"kubernetes.io/projected/140d328c-1945-493d-b208-f89a0bd03b2d-kube-api-access-stlgf\") pod \"maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.328483 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.328412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:53.600186 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.600158    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"]
Sep 18 14:27:53.601637 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:27:53.601603    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140d328c_1945_493d_b208_f89a0bd03b2d.slice/crio-438eb6b7ac4ce3335dc8133b4cddfd3d47768f2644ea30f554ded0ec14750d82 WatchSource:0}: Error finding container 438eb6b7ac4ce3335dc8133b4cddfd3d47768f2644ea30f554ded0ec14750d82: Status 404 returned error can't find the container with id 438eb6b7ac4ce3335dc8133b4cddfd3d47768f2644ea30f554ded0ec14750d82
Sep 18 14:27:53.959907 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:53.959869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9" event={"ID":"140d328c-1945-493d-b208-f89a0bd03b2d","Type":"ContainerStarted","Data":"438eb6b7ac4ce3335dc8133b4cddfd3d47768f2644ea30f554ded0ec14750d82"}
Sep 18 14:27:54.967546 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:54.967498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9" event={"ID":"140d328c-1945-493d-b208-f89a0bd03b2d","Type":"ContainerStarted","Data":"3bdd45e87a6acb8844ae6cf39b07fe513b01e702701c161653caf1179ee1ef4b"}
Sep 18 14:27:54.967935 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:54.967570    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:27:54.984889 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:54.984843    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9" podStartSLOduration=2.602258302 podStartE2EDuration="2.984828875s" totalImagesPullingTime="382.570573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:27:54.982584467 +0000 UTC m=+1763.139104746" watchObservedRunningTime="2026-09-18 14:27:54.984828875 +0000 UTC m=+1763.141349154"
Sep 18 14:27:57.425612 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:27:57.425575    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:28:00.979174 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:00.979109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:28:01.411658 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.411555    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks" podUID="a09d08e2-83e0-4ee6-9b14-65f00cb8c966" containerName="main" containerID="cri-o://75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be" gracePeriod=45
Sep 18 14:28:01.660024 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.659997    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:28:01.737164 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.737117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fqmh\" (UniqueName: \"kubernetes.io/projected/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-kube-api-access-6fqmh\") pod \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") "
Sep 18 14:28:01.737271 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.737173    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-model-cache\") pod \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") "
Sep 18 14:28:01.737271 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.737218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tls-certs\") pod \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") "
Sep 18 14:28:01.737349 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.737270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-dshm\") pod \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") "
Sep 18 14:28:01.737349 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.737298    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-home\") pod \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") "
Sep 18 14:28:01.737349 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.737315    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tmp-dir\") pod \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\" (UID: \"a09d08e2-83e0-4ee6-9b14-65f00cb8c966\") "
Sep 18 14:28:01.737518 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.737471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-model-cache" (OuterVolumeSpecName: "model-cache") pod "a09d08e2-83e0-4ee6-9b14-65f00cb8c966" (UID: "a09d08e2-83e0-4ee6-9b14-65f00cb8c966"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:28:01.737586 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.737569    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-home" (OuterVolumeSpecName: "home") pod "a09d08e2-83e0-4ee6-9b14-65f00cb8c966" (UID: "a09d08e2-83e0-4ee6-9b14-65f00cb8c966"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:28:01.737676 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.737652    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a09d08e2-83e0-4ee6-9b14-65f00cb8c966" (UID: "a09d08e2-83e0-4ee6-9b14-65f00cb8c966"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:28:01.739208 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.739182    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-kube-api-access-6fqmh" (OuterVolumeSpecName: "kube-api-access-6fqmh") pod "a09d08e2-83e0-4ee6-9b14-65f00cb8c966" (UID: "a09d08e2-83e0-4ee6-9b14-65f00cb8c966"). InnerVolumeSpecName "kube-api-access-6fqmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:28:01.739310 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.739272    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-dshm" (OuterVolumeSpecName: "dshm") pod "a09d08e2-83e0-4ee6-9b14-65f00cb8c966" (UID: "a09d08e2-83e0-4ee6-9b14-65f00cb8c966"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:28:01.739464 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.739442    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a09d08e2-83e0-4ee6-9b14-65f00cb8c966" (UID: "a09d08e2-83e0-4ee6-9b14-65f00cb8c966"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:28:01.838404 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.838382    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fqmh\" (UniqueName: \"kubernetes.io/projected/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-kube-api-access-6fqmh\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:01.838404 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.838403    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:01.838525 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.838412    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:01.838525 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.838420    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:01.838525 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.838427    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:01.838525 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:01.838434    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09d08e2-83e0-4ee6-9b14-65f00cb8c966-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:02.009911 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.009852    2563 generic.go:356] "Generic (PLEG): container finished" podID="a09d08e2-83e0-4ee6-9b14-65f00cb8c966" containerID="75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be" exitCode=0
Sep 18 14:28:02.009911 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.009899    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"
Sep 18 14:28:02.010364 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.009940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks" event={"ID":"a09d08e2-83e0-4ee6-9b14-65f00cb8c966","Type":"ContainerDied","Data":"75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be"}
Sep 18 14:28:02.010364 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.009988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks" event={"ID":"a09d08e2-83e0-4ee6-9b14-65f00cb8c966","Type":"ContainerDied","Data":"e09884c73829f97b9b57b0ad09f050c3b98b626962ea66174cb5f3e10c852623"}
Sep 18 14:28:02.010364 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.010006    2563 scope.go:117] "RemoveContainer" containerID="75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be"
Sep 18 14:28:02.020172 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.020152    2563 scope.go:117] "RemoveContainer" containerID="75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be"
Sep 18 14:28:02.020484 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:28:02.020466    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be\": container with ID starting with 75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be not found: ID does not exist" containerID="75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be"
Sep 18 14:28:02.020532 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.020492    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be"} err="failed to get container status \"75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be\": rpc error: code = NotFound desc = could not find container \"75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be\": container with ID starting with 75ea0058a63ed1c4525c8204552197d80796a70158d536b81f06f572078eb2be not found: ID does not exist"
Sep 18 14:28:02.032356 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.032334    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"]
Sep 18 14:28:02.036523 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.036499    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ae54af/rate-test-model-ae54af-kserve-67c65f8dc-5j7ks"]
Sep 18 14:28:02.288631 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.288562    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09d08e2-83e0-4ee6-9b14-65f00cb8c966" path="/var/lib/kubelet/pods/a09d08e2-83e0-4ee6-9b14-65f00cb8c966/volumes"
Sep 18 14:28:02.349867 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.349833    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t" podUID="31c8f7fd-0f89-49e5-8bd8-1e988753b165" containerName="main" containerID="cri-o://62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7" gracePeriod=45
Sep 18 14:28:02.589741 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.589716    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:28:02.645692 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.645665    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tmp-dir\") pod \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") "
Sep 18 14:28:02.645809 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.645714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-model-cache\") pod \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") "
Sep 18 14:28:02.645809 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.645746    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-home\") pod \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") "
Sep 18 14:28:02.645809 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.645764    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tls-certs\") pod \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") "
Sep 18 14:28:02.645968 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.645823    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfsch\" (UniqueName: \"kubernetes.io/projected/31c8f7fd-0f89-49e5-8bd8-1e988753b165-kube-api-access-mfsch\") pod \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") "
Sep 18 14:28:02.645968 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.645853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-dshm\") pod \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\" (UID: \"31c8f7fd-0f89-49e5-8bd8-1e988753b165\") "
Sep 18 14:28:02.645968 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.645949    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31c8f7fd-0f89-49e5-8bd8-1e988753b165" (UID: "31c8f7fd-0f89-49e5-8bd8-1e988753b165"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:28:02.646120 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.645994    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-home" (OuterVolumeSpecName: "home") pod "31c8f7fd-0f89-49e5-8bd8-1e988753b165" (UID: "31c8f7fd-0f89-49e5-8bd8-1e988753b165"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:28:02.646120 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.646015    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-model-cache" (OuterVolumeSpecName: "model-cache") pod "31c8f7fd-0f89-49e5-8bd8-1e988753b165" (UID: "31c8f7fd-0f89-49e5-8bd8-1e988753b165"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:28:02.646120 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.646070    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:02.646120 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.646086    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:02.647919 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.647893    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31c8f7fd-0f89-49e5-8bd8-1e988753b165" (UID: "31c8f7fd-0f89-49e5-8bd8-1e988753b165"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:28:02.648020 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.647918    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-dshm" (OuterVolumeSpecName: "dshm") pod "31c8f7fd-0f89-49e5-8bd8-1e988753b165" (UID: "31c8f7fd-0f89-49e5-8bd8-1e988753b165"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:28:02.648020 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.647963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c8f7fd-0f89-49e5-8bd8-1e988753b165-kube-api-access-mfsch" (OuterVolumeSpecName: "kube-api-access-mfsch") pod "31c8f7fd-0f89-49e5-8bd8-1e988753b165" (UID: "31c8f7fd-0f89-49e5-8bd8-1e988753b165"). InnerVolumeSpecName "kube-api-access-mfsch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:28:02.746880 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.746859    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfsch\" (UniqueName: \"kubernetes.io/projected/31c8f7fd-0f89-49e5-8bd8-1e988753b165-kube-api-access-mfsch\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:02.746880 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.746878    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:02.747017 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.746892    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31c8f7fd-0f89-49e5-8bd8-1e988753b165-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:02.747017 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:02.746900    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31c8f7fd-0f89-49e5-8bd8-1e988753b165-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:28:03.016916 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:03.016888    2563 generic.go:356] "Generic (PLEG): container finished" podID="31c8f7fd-0f89-49e5-8bd8-1e988753b165" containerID="62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7" exitCode=0
Sep 18 14:28:03.017308 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:03.016932    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"
Sep 18 14:28:03.017308 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:03.016955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t" event={"ID":"31c8f7fd-0f89-49e5-8bd8-1e988753b165","Type":"ContainerDied","Data":"62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7"}
Sep 18 14:28:03.017308 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:03.016994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t" event={"ID":"31c8f7fd-0f89-49e5-8bd8-1e988753b165","Type":"ContainerDied","Data":"27662563e18e6da5f5dfd1b1f9158528c21a0d7040042039058d813622b81843"}
Sep 18 14:28:03.017308 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:03.017013    2563 scope.go:117] "RemoveContainer" containerID="62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7"
Sep 18 14:28:03.025955 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:03.025937    2563 scope.go:117] "RemoveContainer" containerID="62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7"
Sep 18 14:28:03.026237 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:28:03.026221    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7\": container with ID starting with 62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7 not found: ID does not exist" containerID="62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7"
Sep 18 14:28:03.026288 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:03.026246    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7"} err="failed to get container status \"62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7\": rpc error: code = NotFound desc = could not find container \"62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7\": container with ID starting with 62b6039c190f9cacc11253532e2e37af93a9f3d97388edc4cd07739e10a398d7 not found: ID does not exist"
Sep 18 14:28:03.038101 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:03.038080    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"]
Sep 18 14:28:03.041821 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:03.041801    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-980661/rate-test-model-980661-kserve-6646f6b955-xds5t"]
Sep 18 14:28:04.288308 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:04.288270    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31c8f7fd-0f89-49e5-8bd8-1e988753b165" path="/var/lib/kubelet/pods/31c8f7fd-0f89-49e5-8bd8-1e988753b165/volumes"
Sep 18 14:28:16.949108 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:16.949071    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"]
Sep 18 14:28:16.949590 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:16.949578    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09d08e2-83e0-4ee6-9b14-65f00cb8c966" containerName="main"
Sep 18 14:28:16.949636 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:16.949593    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09d08e2-83e0-4ee6-9b14-65f00cb8c966" containerName="main"
Sep 18 14:28:16.949636 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:16.949613    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31c8f7fd-0f89-49e5-8bd8-1e988753b165" containerName="main"
Sep 18 14:28:16.949636 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:16.949621    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="31c8f7fd-0f89-49e5-8bd8-1e988753b165" containerName="main"
Sep 18 14:28:16.949734 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:16.949722    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09d08e2-83e0-4ee6-9b14-65f00cb8c966" containerName="main"
Sep 18 14:28:16.949767 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:16.949736    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="31c8f7fd-0f89-49e5-8bd8-1e988753b165" containerName="main"
Sep 18 14:28:16.953348 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:16.953320    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:16.956265 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:16.956240    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 14:28:16.956427 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:16.956403    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7ed186-openshift-default-dockercfg-mq2gl\""
Sep 18 14:28:16.963609 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:16.963582    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"]
Sep 18 14:28:17.055153 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.055115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j7q4\" (UniqueName: \"kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-kube-api-access-7j7q4\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.055256 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.055172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-istio-token\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.055315 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.055258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-certs\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.055315 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.055310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-socket\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.055405 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.055329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f464c4-bde4-4048-afca-b9d96fef0676-istio-podinfo\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.055405 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.055345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-data\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.055405 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.055363    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f464c4-bde4-4048-afca-b9d96fef0676-istiod-ca-cert\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.055405 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.055380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-envoy\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.055532 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.055432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-credential-socket\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156062 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156039    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j7q4\" (UniqueName: \"kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-kube-api-access-7j7q4\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156186 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-istio-token\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156186 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-certs\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156308 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-socket\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156357 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f464c4-bde4-4048-afca-b9d96fef0676-istio-podinfo\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156398 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-data\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156459 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f464c4-bde4-4048-afca-b9d96fef0676-istiod-ca-cert\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156520 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-envoy\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156520 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-certs\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156623 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-credential-socket\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156677 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-socket\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156729 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-data\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.156818 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.156798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-credential-socket\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.157220 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.157198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f464c4-bde4-4048-afca-b9d96fef0676-istiod-ca-cert\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.158700 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.158675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-envoy\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.158806 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.158790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f464c4-bde4-4048-afca-b9d96fef0676-istio-podinfo\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.165952 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.165934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j7q4\" (UniqueName: \"kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-kube-api-access-7j7q4\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.166238 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.166219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-istio-token\") pod \"e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") " pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.266925 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.266874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:17.553808 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.553771    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"]
Sep 18 14:28:17.555050 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:28:17.555017    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f464c4_bde4_4048_afca_b9d96fef0676.slice/crio-28cc5e66bcbdf29fae5942add6421a1967b4fbe90195a9d2a052edb0fcc03c16 WatchSource:0}: Error finding container 28cc5e66bcbdf29fae5942add6421a1967b4fbe90195a9d2a052edb0fcc03c16: Status 404 returned error can't find the container with id 28cc5e66bcbdf29fae5942add6421a1967b4fbe90195a9d2a052edb0fcc03c16
Sep 18 14:28:17.557267 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.557228    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:28:17.557380 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.557311    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:28:17.557380 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:17.557354    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:28:18.113848 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:18.113812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" event={"ID":"84f464c4-bde4-4048-afca-b9d96fef0676","Type":"ContainerStarted","Data":"f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf"}
Sep 18 14:28:18.114276 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:18.113869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" event={"ID":"84f464c4-bde4-4048-afca-b9d96fef0676","Type":"ContainerStarted","Data":"28cc5e66bcbdf29fae5942add6421a1967b4fbe90195a9d2a052edb0fcc03c16"}
Sep 18 14:28:18.267450 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:18.267419    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:18.272358 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:18.272333    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:18.298404 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:18.298356    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" podStartSLOduration=2.2983425840000002 podStartE2EDuration="2.298342584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:28:18.146633696 +0000 UTC m=+1786.303153978" watchObservedRunningTime="2026-09-18 14:28:18.298342584 +0000 UTC m=+1786.454862864"
Sep 18 14:28:19.119162 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:19.119110    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:19.120034 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:19.120013    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:23.005891 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.005843    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"]
Sep 18 14:28:23.010718 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.010687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.013205 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.013183    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7ed186-serving-cert\""
Sep 18 14:28:23.020715 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.020680    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"]
Sep 18 14:28:23.102628 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.102608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd28f\" (UniqueName: \"kubernetes.io/projected/65ba5521-c825-45bd-8620-88d8876e606f-kube-api-access-sd28f\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.102751 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.102647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ba5521-c825-45bd-8620-88d8876e606f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.102751 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.102666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.102751 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.102720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-metrics-certs\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.203445 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.203406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-metrics-certs\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.203580 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.203509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd28f\" (UniqueName: \"kubernetes.io/projected/65ba5521-c825-45bd-8620-88d8876e606f-kube-api-access-sd28f\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.203656 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.203608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ba5521-c825-45bd-8620-88d8876e606f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.203656 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.203634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.203762 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:28:23.203746    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-serving-cert: secret "maas-api-e2e-ipp-b-7ed186-serving-cert" not found
Sep 18 14:28:23.203843 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:28:23.203832    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-maas-api-tls podName:65ba5521-c825-45bd-8620-88d8876e606f nodeName:}" failed. No retries permitted until 2026-09-18 14:28:23.703805959 +0000 UTC m=+1791.860326217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-maas-api-tls") pod "maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz" (UID: "65ba5521-c825-45bd-8620-88d8876e606f") : secret "maas-api-e2e-ipp-b-7ed186-serving-cert" not found
Sep 18 14:28:23.204381 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.204361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ba5521-c825-45bd-8620-88d8876e606f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.205672 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.205656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-metrics-certs\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.211455 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.211434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd28f\" (UniqueName: \"kubernetes.io/projected/65ba5521-c825-45bd-8620-88d8876e606f-kube-api-access-sd28f\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.707637 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.707596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.710047 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.710025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:23.928619 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:23.928588    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:24.204182 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:24.204153    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"]
Sep 18 14:28:24.205511 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:28:24.205457    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ba5521_c825_45bd_8620_88d8876e606f.slice/crio-f4d1eb8c61deb690963c774d9855c927834c4734d57499987448579b7fa9b5b9 WatchSource:0}: Error finding container f4d1eb8c61deb690963c774d9855c927834c4734d57499987448579b7fa9b5b9: Status 404 returned error can't find the container with id f4d1eb8c61deb690963c774d9855c927834c4734d57499987448579b7fa9b5b9
Sep 18 14:28:24.207177 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:24.207155    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:28:25.155616 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:25.155543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz" event={"ID":"65ba5521-c825-45bd-8620-88d8876e606f","Type":"ContainerStarted","Data":"c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3"}
Sep 18 14:28:25.155616 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:25.155601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz" event={"ID":"65ba5521-c825-45bd-8620-88d8876e606f","Type":"ContainerStarted","Data":"f4d1eb8c61deb690963c774d9855c927834c4734d57499987448579b7fa9b5b9"}
Sep 18 14:28:25.155787 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:25.155647    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:25.173039 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:25.172996    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz" podStartSLOduration=2.808194247 podStartE2EDuration="3.172981811s" totalImagesPullingTime="364.787564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:28:25.171037152 +0000 UTC m=+1793.327557435" watchObservedRunningTime="2026-09-18 14:28:25.172981811 +0000 UTC m=+1793.329502130"
Sep 18 14:28:27.720193 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:27.720161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:28:31.165936 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:31.165907    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:28:32.834877 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:32.834849    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:28:32.844863 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:32.844838    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:28:38.231842 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:38.231810    2563 generic.go:356] "Generic (PLEG): container finished" podID="84f464c4-bde4-4048-afca-b9d96fef0676" containerID="f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf" exitCode=0
Sep 18 14:28:38.232265 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:38.231868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" event={"ID":"84f464c4-bde4-4048-afca-b9d96fef0676","Type":"ContainerDied","Data":"f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf"}
Sep 18 14:28:38.232265 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:38.232258    2563 scope.go:117] "RemoveContainer" containerID="f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf"
Sep 18 14:28:38.233265 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:38.233234    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:28:38.233319 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:38.233304    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:28:38.233354 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:38.233336    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236192Ki","pods":"250"}
Sep 18 14:28:39.239831 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:39.239790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" event={"ID":"84f464c4-bde4-4048-afca-b9d96fef0676","Type":"ContainerStarted","Data":"9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441"}
Sep 18 14:28:39.268051 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:39.268023    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:39.268241 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:39.268220    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Sep 18 14:28:39.268289 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:39.268269    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Sep 18 14:28:40.267293 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:40.267257    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Sep 18 14:28:40.267675 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:40.267310    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Sep 18 14:28:41.267333 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:41.267292    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Sep 18 14:28:41.267736 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:41.267359    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Sep 18 14:28:42.267385 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:42.267352    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Sep 18 14:28:42.267778 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:42.267410    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Sep 18 14:28:43.267889 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:43.267854    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Sep 18 14:28:43.268301 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:43.267924    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Sep 18 14:28:44.271178 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:44.271123    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:44.271554 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:44.271386    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:44.272072 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:44.272052    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:28:57.311259 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.311226    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"]
Sep 18 14:28:57.315790 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.315768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.319159 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.319124    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3ab8a5\"/\"kube-root-ca.crt\""
Sep 18 14:28:57.319311 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.319216    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3ab8a5\"/\"default-dockercfg-wb7lh\""
Sep 18 14:28:57.319311 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.319190    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3ab8a5\"/\"openshift-service-ca.crt\""
Sep 18 14:28:57.319311 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.319255    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3ab8a5\"/\"ipp-route-3ab8a5-kserve-self-signed-certs\""
Sep 18 14:28:57.323709 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.323686    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"]
Sep 18 14:28:57.405173 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.405139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-dshm\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.405307 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.405186    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-home\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.405307 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.405213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-tmp-dir\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.405307 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.405251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx8wh\" (UniqueName: \"kubernetes.io/projected/7b0b4682-118b-4714-8341-bebb96b2c285-kube-api-access-jx8wh\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.405307 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.405295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-model-cache\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.405471 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.405326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0b4682-118b-4714-8341-bebb96b2c285-tls-certs\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.506206 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.506175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-model-cache\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.506206 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.506209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0b4682-118b-4714-8341-bebb96b2c285-tls-certs\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.506364 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.506245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-dshm\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.506364 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.506262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-home\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.506364 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.506283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-tmp-dir\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.506364 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.506315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx8wh\" (UniqueName: \"kubernetes.io/projected/7b0b4682-118b-4714-8341-bebb96b2c285-kube-api-access-jx8wh\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.506634 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.506613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-model-cache\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.506708 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.506683    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-tmp-dir\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.506776 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.506752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-home\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.508635 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.508604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-dshm\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.508904 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.508887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0b4682-118b-4714-8341-bebb96b2c285-tls-certs\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.514079 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.514060    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx8wh\" (UniqueName: \"kubernetes.io/projected/7b0b4682-118b-4714-8341-bebb96b2c285-kube-api-access-jx8wh\") pod \"ipp-route-3ab8a5-kserve-5f85cf4987-zg296\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") " pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.626120 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.626066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:28:57.901668 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:57.901635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"]
Sep 18 14:28:57.902980 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:28:57.902955    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b0b4682_118b_4714_8341_bebb96b2c285.slice/crio-533bf7715d284d5e6a46f07745a13bb6fef488498137ee33c9d600e4898f55f1 WatchSource:0}: Error finding container 533bf7715d284d5e6a46f07745a13bb6fef488498137ee33c9d600e4898f55f1: Status 404 returned error can't find the container with id 533bf7715d284d5e6a46f07745a13bb6fef488498137ee33c9d600e4898f55f1
Sep 18 14:28:58.342849 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:58.342818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296" event={"ID":"7b0b4682-118b-4714-8341-bebb96b2c285","Type":"ContainerStarted","Data":"29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c"}
Sep 18 14:28:58.343207 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:58.342855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296" event={"ID":"7b0b4682-118b-4714-8341-bebb96b2c285","Type":"ContainerStarted","Data":"533bf7715d284d5e6a46f07745a13bb6fef488498137ee33c9d600e4898f55f1"}
Sep 18 14:28:58.361929 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:28:58.361884    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296" podStartSLOduration=1.361868617 podStartE2EDuration="1.361868617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:28:58.360537958 +0000 UTC m=+1826.517058242" watchObservedRunningTime="2026-09-18 14:28:58.361868617 +0000 UTC m=+1826.518388943"
Sep 18 14:29:01.793433 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:29:01.793403    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:29:07.626943 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:29:07.626904    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:29:07.627380 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:29:07.626956    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:29:07.639481 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:29:07.639455    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:29:08.413604 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:29:08.413570    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:29:14.720503 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:29:14.720468    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:29:33.924530 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:29:33.924493    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:30:01.053188 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:01.053101    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:30:20.129948 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:20.129908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:30:22.610150 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.610081    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"]
Sep 18 14:30:22.610699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.610379    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9" podUID="140d328c-1945-493d-b208-f89a0bd03b2d" containerName="maas-api" containerID="cri-o://3bdd45e87a6acb8844ae6cf39b07fe513b01e702701c161653caf1179ee1ef4b" gracePeriod=30
Sep 18 14:30:22.828407 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.828375    2563 generic.go:356] "Generic (PLEG): container finished" podID="140d328c-1945-493d-b208-f89a0bd03b2d" containerID="3bdd45e87a6acb8844ae6cf39b07fe513b01e702701c161653caf1179ee1ef4b" exitCode=0
Sep 18 14:30:22.828573 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.828457    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9" event={"ID":"140d328c-1945-493d-b208-f89a0bd03b2d","Type":"ContainerDied","Data":"3bdd45e87a6acb8844ae6cf39b07fe513b01e702701c161653caf1179ee1ef4b"}
Sep 18 14:30:22.870397 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.870338    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:30:22.927702 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.927671    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stlgf\" (UniqueName: \"kubernetes.io/projected/140d328c-1945-493d-b208-f89a0bd03b2d-kube-api-access-stlgf\") pod \"140d328c-1945-493d-b208-f89a0bd03b2d\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") "
Sep 18 14:30:22.927849 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.927727    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-metrics-certs\") pod \"140d328c-1945-493d-b208-f89a0bd03b2d\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") "
Sep 18 14:30:22.927849 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.927786    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/140d328c-1945-493d-b208-f89a0bd03b2d-openshift-service-ca\") pod \"140d328c-1945-493d-b208-f89a0bd03b2d\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") "
Sep 18 14:30:22.927849 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.927823    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-maas-api-tls\") pod \"140d328c-1945-493d-b208-f89a0bd03b2d\" (UID: \"140d328c-1945-493d-b208-f89a0bd03b2d\") "
Sep 18 14:30:22.928224 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.928200    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140d328c-1945-493d-b208-f89a0bd03b2d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "140d328c-1945-493d-b208-f89a0bd03b2d" (UID: "140d328c-1945-493d-b208-f89a0bd03b2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:30:22.929805 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.929780    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "140d328c-1945-493d-b208-f89a0bd03b2d" (UID: "140d328c-1945-493d-b208-f89a0bd03b2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:30:22.929904 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.929855    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140d328c-1945-493d-b208-f89a0bd03b2d-kube-api-access-stlgf" (OuterVolumeSpecName: "kube-api-access-stlgf") pod "140d328c-1945-493d-b208-f89a0bd03b2d" (UID: "140d328c-1945-493d-b208-f89a0bd03b2d"). InnerVolumeSpecName "kube-api-access-stlgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:30:22.929945 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:22.929905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "140d328c-1945-493d-b208-f89a0bd03b2d" (UID: "140d328c-1945-493d-b208-f89a0bd03b2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:30:23.029401 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:23.029370    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stlgf\" (UniqueName: \"kubernetes.io/projected/140d328c-1945-493d-b208-f89a0bd03b2d-kube-api-access-stlgf\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:30:23.029401 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:23.029398    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-metrics-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:30:23.029551 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:23.029408    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/140d328c-1945-493d-b208-f89a0bd03b2d-openshift-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:30:23.029551 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:23.029419    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/140d328c-1945-493d-b208-f89a0bd03b2d-maas-api-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:30:23.835938 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:23.835908    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"
Sep 18 14:30:23.836399 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:23.835902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9" event={"ID":"140d328c-1945-493d-b208-f89a0bd03b2d","Type":"ContainerDied","Data":"438eb6b7ac4ce3335dc8133b4cddfd3d47768f2644ea30f554ded0ec14750d82"}
Sep 18 14:30:23.836399 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:23.836050    2563 scope.go:117] "RemoveContainer" containerID="3bdd45e87a6acb8844ae6cf39b07fe513b01e702701c161653caf1179ee1ef4b"
Sep 18 14:30:23.865400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:23.865369    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"]
Sep 18 14:30:23.867813 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:23.867789    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3ab8a5-757cc86469-sl2f9"]
Sep 18 14:30:24.288607 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:24.288576    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="140d328c-1945-493d-b208-f89a0bd03b2d" path="/var/lib/kubelet/pods/140d328c-1945-493d-b208-f89a0bd03b2d/volumes"
Sep 18 14:30:28.196049 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:28.196011    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"]
Sep 18 14:30:33.610469 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:33.610425    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:30:43.216275 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.216234    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296" podUID="7b0b4682-118b-4714-8341-bebb96b2c285" containerName="main" containerID="cri-o://29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c" gracePeriod=45
Sep 18 14:30:43.468480 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.468415    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:30:43.604322 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.604282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx8wh\" (UniqueName: \"kubernetes.io/projected/7b0b4682-118b-4714-8341-bebb96b2c285-kube-api-access-jx8wh\") pod \"7b0b4682-118b-4714-8341-bebb96b2c285\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") "
Sep 18 14:30:43.604523 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.604340    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0b4682-118b-4714-8341-bebb96b2c285-tls-certs\") pod \"7b0b4682-118b-4714-8341-bebb96b2c285\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") "
Sep 18 14:30:43.604523 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.604417    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-model-cache\") pod \"7b0b4682-118b-4714-8341-bebb96b2c285\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") "
Sep 18 14:30:43.604523 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.604446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-home\") pod \"7b0b4682-118b-4714-8341-bebb96b2c285\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") "
Sep 18 14:30:43.604523 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.604477    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-dshm\") pod \"7b0b4682-118b-4714-8341-bebb96b2c285\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") "
Sep 18 14:30:43.604523 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.604503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-tmp-dir\") pod \"7b0b4682-118b-4714-8341-bebb96b2c285\" (UID: \"7b0b4682-118b-4714-8341-bebb96b2c285\") "
Sep 18 14:30:43.604799 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.604759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b0b4682-118b-4714-8341-bebb96b2c285" (UID: "7b0b4682-118b-4714-8341-bebb96b2c285"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:30:43.604799 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.604752    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-home" (OuterVolumeSpecName: "home") pod "7b0b4682-118b-4714-8341-bebb96b2c285" (UID: "7b0b4682-118b-4714-8341-bebb96b2c285"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:30:43.604894 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.604806    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b0b4682-118b-4714-8341-bebb96b2c285" (UID: "7b0b4682-118b-4714-8341-bebb96b2c285"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:30:43.606701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.606675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0b4682-118b-4714-8341-bebb96b2c285-kube-api-access-jx8wh" (OuterVolumeSpecName: "kube-api-access-jx8wh") pod "7b0b4682-118b-4714-8341-bebb96b2c285" (UID: "7b0b4682-118b-4714-8341-bebb96b2c285"). InnerVolumeSpecName "kube-api-access-jx8wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:30:43.606701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.606694    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-dshm" (OuterVolumeSpecName: "dshm") pod "7b0b4682-118b-4714-8341-bebb96b2c285" (UID: "7b0b4682-118b-4714-8341-bebb96b2c285"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:30:43.606854 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.606707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0b4682-118b-4714-8341-bebb96b2c285-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b0b4682-118b-4714-8341-bebb96b2c285" (UID: "7b0b4682-118b-4714-8341-bebb96b2c285"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:30:43.705301 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.705267    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx8wh\" (UniqueName: \"kubernetes.io/projected/7b0b4682-118b-4714-8341-bebb96b2c285-kube-api-access-jx8wh\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:30:43.705301 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.705296    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b0b4682-118b-4714-8341-bebb96b2c285-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:30:43.705301 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.705306    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:30:43.705528 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.705314    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:30:43.705528 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.705321    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:30:43.705528 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.705328    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b0b4682-118b-4714-8341-bebb96b2c285-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:30:43.965000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.964954    2563 generic.go:356] "Generic (PLEG): container finished" podID="7b0b4682-118b-4714-8341-bebb96b2c285" containerID="29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c" exitCode=0
Sep 18 14:30:43.965000 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.964999    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"
Sep 18 14:30:43.965277 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.965038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296" event={"ID":"7b0b4682-118b-4714-8341-bebb96b2c285","Type":"ContainerDied","Data":"29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c"}
Sep 18 14:30:43.965277 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.965088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296" event={"ID":"7b0b4682-118b-4714-8341-bebb96b2c285","Type":"ContainerDied","Data":"533bf7715d284d5e6a46f07745a13bb6fef488498137ee33c9d600e4898f55f1"}
Sep 18 14:30:43.965277 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.965107    2563 scope.go:117] "RemoveContainer" containerID="29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c"
Sep 18 14:30:43.974906 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.974887    2563 scope.go:117] "RemoveContainer" containerID="29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c"
Sep 18 14:30:43.975192 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:30:43.975173    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c\": container with ID starting with 29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c not found: ID does not exist" containerID="29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c"
Sep 18 14:30:43.975243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.975201    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c"} err="failed to get container status \"29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c\": rpc error: code = NotFound desc = could not find container \"29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c\": container with ID starting with 29995c639f06e1ae80705432ab288a0bf14b8b55de0a40dc1960d5826763642c not found: ID does not exist"
Sep 18 14:30:43.986817 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.986793    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"]
Sep 18 14:30:43.991906 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:43.991884    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3ab8a5/ipp-route-3ab8a5-kserve-5f85cf4987-zg296"]
Sep 18 14:30:44.288875 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:30:44.288815    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b0b4682-118b-4714-8341-bebb96b2c285" path="/var/lib/kubelet/pods/7b0b4682-118b-4714-8341-bebb96b2c285/volumes"
Sep 18 14:31:37.735029 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:37.734992    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"]
Sep 18 14:31:37.735460 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:37.735194    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz" podUID="65ba5521-c825-45bd-8620-88d8876e606f" containerName="maas-api" containerID="cri-o://c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3" gracePeriod=30
Sep 18 14:31:37.983836 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:37.983812    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:31:38.141801 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.141719    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ba5521-c825-45bd-8620-88d8876e606f-openshift-service-ca\") pod \"65ba5521-c825-45bd-8620-88d8876e606f\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") "
Sep 18 14:31:38.141801 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.141785    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-maas-api-tls\") pod \"65ba5521-c825-45bd-8620-88d8876e606f\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") "
Sep 18 14:31:38.142013 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.141815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd28f\" (UniqueName: \"kubernetes.io/projected/65ba5521-c825-45bd-8620-88d8876e606f-kube-api-access-sd28f\") pod \"65ba5521-c825-45bd-8620-88d8876e606f\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") "
Sep 18 14:31:38.142013 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.141849    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-metrics-certs\") pod \"65ba5521-c825-45bd-8620-88d8876e606f\" (UID: \"65ba5521-c825-45bd-8620-88d8876e606f\") "
Sep 18 14:31:38.142152 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.142110    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ba5521-c825-45bd-8620-88d8876e606f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65ba5521-c825-45bd-8620-88d8876e606f" (UID: "65ba5521-c825-45bd-8620-88d8876e606f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:31:38.143971 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.143936    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "65ba5521-c825-45bd-8620-88d8876e606f" (UID: "65ba5521-c825-45bd-8620-88d8876e606f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:31:38.144086 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.143993    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65ba5521-c825-45bd-8620-88d8876e606f" (UID: "65ba5521-c825-45bd-8620-88d8876e606f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:31:38.144086 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.144019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ba5521-c825-45bd-8620-88d8876e606f-kube-api-access-sd28f" (OuterVolumeSpecName: "kube-api-access-sd28f") pod "65ba5521-c825-45bd-8620-88d8876e606f" (UID: "65ba5521-c825-45bd-8620-88d8876e606f"). InnerVolumeSpecName "kube-api-access-sd28f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:31:38.243220 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.243192    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ba5521-c825-45bd-8620-88d8876e606f-openshift-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:38.243220 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.243218    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-maas-api-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:38.243382 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.243229    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd28f\" (UniqueName: \"kubernetes.io/projected/65ba5521-c825-45bd-8620-88d8876e606f-kube-api-access-sd28f\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:38.243382 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.243237    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65ba5521-c825-45bd-8620-88d8876e606f-metrics-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:38.286179 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.286149    2563 generic.go:356] "Generic (PLEG): container finished" podID="65ba5521-c825-45bd-8620-88d8876e606f" containerID="c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3" exitCode=0
Sep 18 14:31:38.286308 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.286242    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"
Sep 18 14:31:38.289094 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.289068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz" event={"ID":"65ba5521-c825-45bd-8620-88d8876e606f","Type":"ContainerDied","Data":"c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3"}
Sep 18 14:31:38.289242 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.289111    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz" event={"ID":"65ba5521-c825-45bd-8620-88d8876e606f","Type":"ContainerDied","Data":"f4d1eb8c61deb690963c774d9855c927834c4734d57499987448579b7fa9b5b9"}
Sep 18 14:31:38.289242 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.289168    2563 scope.go:117] "RemoveContainer" containerID="c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3"
Sep 18 14:31:38.299454 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.299438    2563 scope.go:117] "RemoveContainer" containerID="c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3"
Sep 18 14:31:38.299702 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:31:38.299684    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3\": container with ID starting with c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3 not found: ID does not exist" containerID="c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3"
Sep 18 14:31:38.299751 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.299712    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3"} err="failed to get container status \"c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3\": rpc error: code = NotFound desc = could not find container \"c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3\": container with ID starting with c6472694289cf253bb43f97fa403946b42848670b1b5123ca5354252a23f49e3 not found: ID does not exist"
Sep 18 14:31:38.315184 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.315164    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"]
Sep 18 14:31:38.322521 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:38.322499    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7ed186-7fb5fbcf-6xdtz"]
Sep 18 14:31:40.287834 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:40.287798    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65ba5521-c825-45bd-8620-88d8876e606f" path="/var/lib/kubelet/pods/65ba5521-c825-45bd-8620-88d8876e606f/volumes"
Sep 18 14:31:40.947895 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:40.947859    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:31:45.613879 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:45.613835    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"]
Sep 18 14:31:45.614335 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:45.614101    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy" containerID="cri-o://9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441" gracePeriod=30
Sep 18 14:31:50.857172 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.857123    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:31:50.947769 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.947742    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-istio-token\") pod \"84f464c4-bde4-4048-afca-b9d96fef0676\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") "
Sep 18 14:31:50.947887 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.947778    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-socket\") pod \"84f464c4-bde4-4048-afca-b9d96fef0676\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") "
Sep 18 14:31:50.947887 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.947805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-envoy\") pod \"84f464c4-bde4-4048-afca-b9d96fef0676\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") "
Sep 18 14:31:50.947887 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.947826    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-credential-socket\") pod \"84f464c4-bde4-4048-afca-b9d96fef0676\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") "
Sep 18 14:31:50.947887 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.947854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-data\") pod \"84f464c4-bde4-4048-afca-b9d96fef0676\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") "
Sep 18 14:31:50.948096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.947893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j7q4\" (UniqueName: \"kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-kube-api-access-7j7q4\") pod \"84f464c4-bde4-4048-afca-b9d96fef0676\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") "
Sep 18 14:31:50.948096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.947925    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-certs\") pod \"84f464c4-bde4-4048-afca-b9d96fef0676\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") "
Sep 18 14:31:50.948096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.947960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f464c4-bde4-4048-afca-b9d96fef0676-istio-podinfo\") pod \"84f464c4-bde4-4048-afca-b9d96fef0676\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") "
Sep 18 14:31:50.948096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.947982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f464c4-bde4-4048-afca-b9d96fef0676-istiod-ca-cert\") pod \"84f464c4-bde4-4048-afca-b9d96fef0676\" (UID: \"84f464c4-bde4-4048-afca-b9d96fef0676\") "
Sep 18 14:31:50.948096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.948047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "84f464c4-bde4-4048-afca-b9d96fef0676" (UID: "84f464c4-bde4-4048-afca-b9d96fef0676"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:31:50.948367 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.948187    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-data" (OuterVolumeSpecName: "istio-data") pod "84f464c4-bde4-4048-afca-b9d96fef0676" (UID: "84f464c4-bde4-4048-afca-b9d96fef0676"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:31:50.948367 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.948204    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "84f464c4-bde4-4048-afca-b9d96fef0676" (UID: "84f464c4-bde4-4048-afca-b9d96fef0676"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:31:50.948367 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.948295    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "84f464c4-bde4-4048-afca-b9d96fef0676" (UID: "84f464c4-bde4-4048-afca-b9d96fef0676"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:31:50.948475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.948410    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:50.948475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.948428    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-credential-socket\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:50.948475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.948442    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-data\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:50.948475 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.948455    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-workload-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:50.948616 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.948525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f464c4-bde4-4048-afca-b9d96fef0676-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "84f464c4-bde4-4048-afca-b9d96fef0676" (UID: "84f464c4-bde4-4048-afca-b9d96fef0676"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:31:50.950223 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.950190    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-kube-api-access-7j7q4" (OuterVolumeSpecName: "kube-api-access-7j7q4") pod "84f464c4-bde4-4048-afca-b9d96fef0676" (UID: "84f464c4-bde4-4048-afca-b9d96fef0676"). InnerVolumeSpecName "kube-api-access-7j7q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:31:50.950325 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.950295    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84f464c4-bde4-4048-afca-b9d96fef0676-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "84f464c4-bde4-4048-afca-b9d96fef0676" (UID: "84f464c4-bde4-4048-afca-b9d96fef0676"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 14:31:50.950457 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.950357    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "84f464c4-bde4-4048-afca-b9d96fef0676" (UID: "84f464c4-bde4-4048-afca-b9d96fef0676"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:31:50.950457 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:50.950377    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-istio-token" (OuterVolumeSpecName: "istio-token") pod "84f464c4-bde4-4048-afca-b9d96fef0676" (UID: "84f464c4-bde4-4048-afca-b9d96fef0676"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:31:51.048774 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.048756    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j7q4\" (UniqueName: \"kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-kube-api-access-7j7q4\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:51.048774 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.048775    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f464c4-bde4-4048-afca-b9d96fef0676-istio-podinfo\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:51.048905 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.048783    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f464c4-bde4-4048-afca-b9d96fef0676-istiod-ca-cert\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:51.048905 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.048793    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f464c4-bde4-4048-afca-b9d96fef0676-istio-token\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:51.048905 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.048801    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f464c4-bde4-4048-afca-b9d96fef0676-istio-envoy\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:51.364763 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.364696    2563 generic.go:356] "Generic (PLEG): container finished" podID="84f464c4-bde4-4048-afca-b9d96fef0676" containerID="9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441" exitCode=0
Sep 18 14:31:51.364883 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.364762    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"
Sep 18 14:31:51.364883 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.364789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" event={"ID":"84f464c4-bde4-4048-afca-b9d96fef0676","Type":"ContainerDied","Data":"9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441"}
Sep 18 14:31:51.364883 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.364856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl" event={"ID":"84f464c4-bde4-4048-afca-b9d96fef0676","Type":"ContainerDied","Data":"28cc5e66bcbdf29fae5942add6421a1967b4fbe90195a9d2a052edb0fcc03c16"}
Sep 18 14:31:51.365047 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.364883    2563 scope.go:117] "RemoveContainer" containerID="9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441"
Sep 18 14:31:51.375207 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.375186    2563 scope.go:117] "RemoveContainer" containerID="f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf"
Sep 18 14:31:51.386745 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.386727    2563 scope.go:117] "RemoveContainer" containerID="9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441"
Sep 18 14:31:51.387045 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:31:51.387023    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441\": container with ID starting with 9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441 not found: ID does not exist" containerID="9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441"
Sep 18 14:31:51.387109 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.387056    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441"} err="failed to get container status \"9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441\": rpc error: code = NotFound desc = could not find container \"9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441\": container with ID starting with 9cefe5026b53ccff024f86e9dd2e32f7ae074b80b758567367cdc89a7caaf441 not found: ID does not exist"
Sep 18 14:31:51.387109 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.387080    2563 scope.go:117] "RemoveContainer" containerID="f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf"
Sep 18 14:31:51.387435 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:31:51.387410    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf\": container with ID starting with f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf not found: ID does not exist" containerID="f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf"
Sep 18 14:31:51.387482 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.387445    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf"} err="failed to get container status \"f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf\": rpc error: code = NotFound desc = could not find container \"f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf\": container with ID starting with f63f6632725e55bfbf7c281897c12ac913c65b1418fd63a6a036f786b2982acf not found: ID does not exist"
Sep 18 14:31:51.402572 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.402544    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"]
Sep 18 14:31:51.425139 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:51.425105    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7ed186-openshift-default-58dd8f6cdb-tjqsl"]
Sep 18 14:31:52.288930 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:52.288897    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" path="/var/lib/kubelet/pods/84f464c4-bde4-4048-afca-b9d96fef0676/volumes"
Sep 18 14:31:56.477791 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.477699    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"]
Sep 18 14:31:56.478619 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.478571    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk" podUID="7b414230-c641-4521-b5e1-03f285fedbef" containerName="maas-api" containerID="cri-o://61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729" gracePeriod=30
Sep 18 14:31:56.716607 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.716577    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:31:56.893638 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.893509    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b414230-c641-4521-b5e1-03f285fedbef-openshift-service-ca\") pod \"7b414230-c641-4521-b5e1-03f285fedbef\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") "
Sep 18 14:31:56.893771 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.893714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-maas-api-tls\") pod \"7b414230-c641-4521-b5e1-03f285fedbef\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") "
Sep 18 14:31:56.893771 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.893753    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b414230-c641-4521-b5e1-03f285fedbef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b414230-c641-4521-b5e1-03f285fedbef" (UID: "7b414230-c641-4521-b5e1-03f285fedbef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:31:56.893771 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.893761    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qnlk\" (UniqueName: \"kubernetes.io/projected/7b414230-c641-4521-b5e1-03f285fedbef-kube-api-access-9qnlk\") pod \"7b414230-c641-4521-b5e1-03f285fedbef\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") "
Sep 18 14:31:56.893948 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.893804    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-metrics-certs\") pod \"7b414230-c641-4521-b5e1-03f285fedbef\" (UID: \"7b414230-c641-4521-b5e1-03f285fedbef\") "
Sep 18 14:31:56.894070 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.894055    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b414230-c641-4521-b5e1-03f285fedbef-openshift-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:56.895730 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.895706    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b414230-c641-4521-b5e1-03f285fedbef-kube-api-access-9qnlk" (OuterVolumeSpecName: "kube-api-access-9qnlk") pod "7b414230-c641-4521-b5e1-03f285fedbef" (UID: "7b414230-c641-4521-b5e1-03f285fedbef"). InnerVolumeSpecName "kube-api-access-9qnlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:31:56.895843 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.895829    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7b414230-c641-4521-b5e1-03f285fedbef" (UID: "7b414230-c641-4521-b5e1-03f285fedbef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:31:56.895928 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.895914    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b414230-c641-4521-b5e1-03f285fedbef" (UID: "7b414230-c641-4521-b5e1-03f285fedbef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:31:56.995370 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.995348    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qnlk\" (UniqueName: \"kubernetes.io/projected/7b414230-c641-4521-b5e1-03f285fedbef-kube-api-access-9qnlk\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:56.995370 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.995367    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-metrics-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:56.995494 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:56.995376    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b414230-c641-4521-b5e1-03f285fedbef-maas-api-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:31:57.406880 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:57.406850    2563 generic.go:356] "Generic (PLEG): container finished" podID="7b414230-c641-4521-b5e1-03f285fedbef" containerID="61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729" exitCode=0
Sep 18 14:31:57.407044 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:57.406889    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"
Sep 18 14:31:57.407044 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:57.406938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk" event={"ID":"7b414230-c641-4521-b5e1-03f285fedbef","Type":"ContainerDied","Data":"61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729"}
Sep 18 14:31:57.407044 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:57.406999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk" event={"ID":"7b414230-c641-4521-b5e1-03f285fedbef","Type":"ContainerDied","Data":"a1277c40b564362c18ece59245b65c28fae2f8edd7dbc618d9891a84df67a0cf"}
Sep 18 14:31:57.407044 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:57.407030    2563 scope.go:117] "RemoveContainer" containerID="61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729"
Sep 18 14:31:57.417151 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:57.417112    2563 scope.go:117] "RemoveContainer" containerID="61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729"
Sep 18 14:31:57.417413 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:31:57.417396    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729\": container with ID starting with 61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729 not found: ID does not exist" containerID="61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729"
Sep 18 14:31:57.417466 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:57.417421    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729"} err="failed to get container status \"61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729\": rpc error: code = NotFound desc = could not find container \"61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729\": container with ID starting with 61823a55e0d7478d3c54fe757bdd0e454a13febc90ac8ba812b71875f937f729 not found: ID does not exist"
Sep 18 14:31:57.434921 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:57.434895    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"]
Sep 18 14:31:57.437122 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:57.437100    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-ae54af-5bb5d67f4d-9tpjk"]
Sep 18 14:31:58.289093 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:58.289049    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b414230-c641-4521-b5e1-03f285fedbef" path="/var/lib/kubelet/pods/7b414230-c641-4521-b5e1-03f285fedbef/volumes"
Sep 18 14:31:59.426975 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:31:59.426939    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:32:15.027277 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.027238    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"]
Sep 18 14:32:15.027707 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.027455    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb" podUID="a445d889-bf7f-4451-89d7-f0e0bc4345e3" containerName="maas-api" containerID="cri-o://0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468" gracePeriod=30
Sep 18 14:32:15.269471 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.269447    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:32:15.337558 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.337491    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-maas-api-tls\") pod \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") "
Sep 18 14:32:15.337682 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.337587    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-metrics-certs\") pod \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") "
Sep 18 14:32:15.337682 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.337625    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm94d\" (UniqueName: \"kubernetes.io/projected/a445d889-bf7f-4451-89d7-f0e0bc4345e3-kube-api-access-gm94d\") pod \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") "
Sep 18 14:32:15.337682 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.337649    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a445d889-bf7f-4451-89d7-f0e0bc4345e3-openshift-service-ca\") pod \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\" (UID: \"a445d889-bf7f-4451-89d7-f0e0bc4345e3\") "
Sep 18 14:32:15.338041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.338014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a445d889-bf7f-4451-89d7-f0e0bc4345e3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a445d889-bf7f-4451-89d7-f0e0bc4345e3" (UID: "a445d889-bf7f-4451-89d7-f0e0bc4345e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:32:15.339545 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.339517    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a445d889-bf7f-4451-89d7-f0e0bc4345e3" (UID: "a445d889-bf7f-4451-89d7-f0e0bc4345e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:32:15.339642 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.339589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a445d889-bf7f-4451-89d7-f0e0bc4345e3-kube-api-access-gm94d" (OuterVolumeSpecName: "kube-api-access-gm94d") pod "a445d889-bf7f-4451-89d7-f0e0bc4345e3" (UID: "a445d889-bf7f-4451-89d7-f0e0bc4345e3"). InnerVolumeSpecName "kube-api-access-gm94d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:32:15.339704 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.339652    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a445d889-bf7f-4451-89d7-f0e0bc4345e3" (UID: "a445d889-bf7f-4451-89d7-f0e0bc4345e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:32:15.438976 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.438949    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-metrics-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:32:15.439070 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.438975    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm94d\" (UniqueName: \"kubernetes.io/projected/a445d889-bf7f-4451-89d7-f0e0bc4345e3-kube-api-access-gm94d\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:32:15.439070 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.438990    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a445d889-bf7f-4451-89d7-f0e0bc4345e3-openshift-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:32:15.439070 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.439002    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a445d889-bf7f-4451-89d7-f0e0bc4345e3-maas-api-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:32:15.508971 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.508946    2563 generic.go:356] "Generic (PLEG): container finished" podID="a445d889-bf7f-4451-89d7-f0e0bc4345e3" containerID="0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468" exitCode=0
Sep 18 14:32:15.509054 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.508988    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"
Sep 18 14:32:15.509054 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.509009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb" event={"ID":"a445d889-bf7f-4451-89d7-f0e0bc4345e3","Type":"ContainerDied","Data":"0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468"}
Sep 18 14:32:15.509054 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.509045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb" event={"ID":"a445d889-bf7f-4451-89d7-f0e0bc4345e3","Type":"ContainerDied","Data":"ce940721d21907a65ee2b8f422c81e2dc260350dc9681e15f63f91d3f38f9314"}
Sep 18 14:32:15.509173 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.509066    2563 scope.go:117] "RemoveContainer" containerID="0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468"
Sep 18 14:32:15.519286 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.519266    2563 scope.go:117] "RemoveContainer" containerID="0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468"
Sep 18 14:32:15.519545 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:32:15.519524    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468\": container with ID starting with 0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468 not found: ID does not exist" containerID="0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468"
Sep 18 14:32:15.519611 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.519558    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468"} err="failed to get container status \"0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468\": rpc error: code = NotFound desc = could not find container \"0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468\": container with ID starting with 0be1cef95c064c56cee36eb08e20b3797f0069ba1243faeb1992e6771ea2b468 not found: ID does not exist"
Sep 18 14:32:15.557694 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.557668    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"]
Sep 18 14:32:15.566033 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:15.566013    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-980661-6b676f9945-cdbpb"]
Sep 18 14:32:16.287561 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:16.287526    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a445d889-bf7f-4451-89d7-f0e0bc4345e3" path="/var/lib/kubelet/pods/a445d889-bf7f-4451-89d7-f0e0bc4345e3/volumes"
Sep 18 14:32:17.618445 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:17.618408    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:32:32.531615 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:32:32.531572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:33:32.874225 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:33:32.874199    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:33:32.883229 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:33:32.883209    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:33:55.016523 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:33:55.016479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:34:34.316799 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:34:34.316763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:34:42.026343 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:34:42.026306    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:34:50.238050 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:34:50.238008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:35:08.114834 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:35:08.114789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:35:14.313036 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:35:14.312998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:36:41.073375 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:41.073336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:36:48.011293 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:48.011251    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:36:56.087255 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087211    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"]
Sep 18 14:36:56.087699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087634    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ba5521-c825-45bd-8620-88d8876e606f" containerName="maas-api"
Sep 18 14:36:56.087699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087654    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ba5521-c825-45bd-8620-88d8876e606f" containerName="maas-api"
Sep 18 14:36:56.087699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087667    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy"
Sep 18 14:36:56.087699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087674    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy"
Sep 18 14:36:56.087699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087686    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0b4682-118b-4714-8341-bebb96b2c285" containerName="main"
Sep 18 14:36:56.087699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087692    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0b4682-118b-4714-8341-bebb96b2c285" containerName="main"
Sep 18 14:36:56.087699 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087702    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a445d889-bf7f-4451-89d7-f0e0bc4345e3" containerName="maas-api"
Sep 18 14:36:56.087937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087707    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a445d889-bf7f-4451-89d7-f0e0bc4345e3" containerName="maas-api"
Sep 18 14:36:56.087937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087725    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140d328c-1945-493d-b208-f89a0bd03b2d" containerName="maas-api"
Sep 18 14:36:56.087937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087731    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="140d328c-1945-493d-b208-f89a0bd03b2d" containerName="maas-api"
Sep 18 14:36:56.087937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087741    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b414230-c641-4521-b5e1-03f285fedbef" containerName="maas-api"
Sep 18 14:36:56.087937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087747    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b414230-c641-4521-b5e1-03f285fedbef" containerName="maas-api"
Sep 18 14:36:56.087937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087818    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0b4682-118b-4714-8341-bebb96b2c285" containerName="main"
Sep 18 14:36:56.087937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087829    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a445d889-bf7f-4451-89d7-f0e0bc4345e3" containerName="maas-api"
Sep 18 14:36:56.087937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087837    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy"
Sep 18 14:36:56.087937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087845    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b414230-c641-4521-b5e1-03f285fedbef" containerName="maas-api"
Sep 18 14:36:56.087937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087851    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="140d328c-1945-493d-b208-f89a0bd03b2d" containerName="maas-api"
Sep 18 14:36:56.087937 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.087858    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ba5521-c825-45bd-8620-88d8876e606f" containerName="maas-api"
Sep 18 14:36:56.091075 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.091044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.096995 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.096969    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-355c1f75-kserve-self-signed-certs\""
Sep 18 14:36:56.117550 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.117514    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"]
Sep 18 14:36:56.171174 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.171123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-home\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.171297 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.171224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-model-cache\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.171297 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.171268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e779c16-c8ed-4229-99c0-4202f7684cb0-tls-certs\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.171417 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.171334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mnl8\" (UniqueName: \"kubernetes.io/projected/6e779c16-c8ed-4229-99c0-4202f7684cb0-kube-api-access-4mnl8\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.171417 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.171400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-tmp-dir\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.171510 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.171425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-dshm\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.272257 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.272228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-model-cache\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.272385 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.272265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e779c16-c8ed-4229-99c0-4202f7684cb0-tls-certs\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.272385 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.272317    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mnl8\" (UniqueName: \"kubernetes.io/projected/6e779c16-c8ed-4229-99c0-4202f7684cb0-kube-api-access-4mnl8\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.272506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.272383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-tmp-dir\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.272506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.272407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-dshm\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.272506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.272442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-home\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.272674 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.272650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-model-cache\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.272736 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.272717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-tmp-dir\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.272865 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.272828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-home\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.274691 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.274658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-dshm\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.274959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.274942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e779c16-c8ed-4229-99c0-4202f7684cb0-tls-certs\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.282225 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.282200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mnl8\" (UniqueName: \"kubernetes.io/projected/6e779c16-c8ed-4229-99c0-4202f7684cb0-kube-api-access-4mnl8\") pod \"e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") " pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.400348 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.400293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:36:56.694004 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:56.693967    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"]
Sep 18 14:36:56.701550 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:36:56.701510    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e779c16_c8ed_4229_99c0_4202f7684cb0.slice/crio-d1150a80a5e2bcd4fd6dd0016021dce9d59e60dbbcaf8a78466dd5d02f9cc438 WatchSource:0}: Error finding container d1150a80a5e2bcd4fd6dd0016021dce9d59e60dbbcaf8a78466dd5d02f9cc438: Status 404 returned error can't find the container with id d1150a80a5e2bcd4fd6dd0016021dce9d59e60dbbcaf8a78466dd5d02f9cc438
Sep 18 14:36:57.100259 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:57.100178    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps" event={"ID":"6e779c16-c8ed-4229-99c0-4202f7684cb0","Type":"ContainerStarted","Data":"2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4"}
Sep 18 14:36:57.100259 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:57.100223    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps" event={"ID":"6e779c16-c8ed-4229-99c0-4202f7684cb0","Type":"ContainerStarted","Data":"d1150a80a5e2bcd4fd6dd0016021dce9d59e60dbbcaf8a78466dd5d02f9cc438"}
Sep 18 14:36:57.134768 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:57.134716    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps" podStartSLOduration=1.134700989 podStartE2EDuration="1.134700989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:36:57.130408413 +0000 UTC m=+2305.286928692" watchObservedRunningTime="2026-09-18 14:36:57.134700989 +0000 UTC m=+2305.291221299"
Sep 18 14:36:58.978377 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:36:58.978335    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:37:06.400566 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:06.400535    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:37:06.400953 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:06.400776    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:37:06.412544 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:06.412518    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:37:07.168044 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:07.168009    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:37:14.977033 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:14.976992    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"]
Sep 18 14:37:14.977506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:14.977374    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy"
Sep 18 14:37:14.977506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:14.977387    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy"
Sep 18 14:37:14.977506 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:14.977466    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f464c4-bde4-4048-afca-b9d96fef0676" containerName="istio-proxy"
Sep 18 14:37:14.980417 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:14.980399    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.001837 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.001808    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-355c1f75-kserve-self-signed-certs\""
Sep 18 14:37:15.021277 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.021255    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"]
Sep 18 14:37:15.121812 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.121787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-home\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.121929 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.121815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-model-cache\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.121929 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.121852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-dshm\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.121929 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.121907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2413cf32-f7ce-4f80-8529-e54027a20a66-tls-certs\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.122152 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.121992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-tmp-dir\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.122152 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.122046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfvz\" (UniqueName: \"kubernetes.io/projected/2413cf32-f7ce-4f80-8529-e54027a20a66-kube-api-access-mdfvz\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.222574 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.222550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-home\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.222689 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.222578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-model-cache\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.222689 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.222614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-dshm\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.222689 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.222641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2413cf32-f7ce-4f80-8529-e54027a20a66-tls-certs\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.222689 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.222669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-tmp-dir\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.222689 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.222686    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfvz\" (UniqueName: \"kubernetes.io/projected/2413cf32-f7ce-4f80-8529-e54027a20a66-kube-api-access-mdfvz\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.223045 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.223019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-model-cache\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.223103 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.223067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-home\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.223161 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.223145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-tmp-dir\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.225043 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.225020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-dshm\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.225377 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.225356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2413cf32-f7ce-4f80-8529-e54027a20a66-tls-certs\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.244760 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.244696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfvz\" (UniqueName: \"kubernetes.io/projected/2413cf32-f7ce-4f80-8529-e54027a20a66-kube-api-access-mdfvz\") pod \"e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") " pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.289998 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.289973    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:15.576243 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:15.576216    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"]
Sep 18 14:37:15.578282 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:37:15.578254    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2413cf32_f7ce_4f80_8529_e54027a20a66.slice/crio-5fe3d2b81618b677129de8623854983affd82acd16c14507d614a78bad60966b WatchSource:0}: Error finding container 5fe3d2b81618b677129de8623854983affd82acd16c14507d614a78bad60966b: Status 404 returned error can't find the container with id 5fe3d2b81618b677129de8623854983affd82acd16c14507d614a78bad60966b
Sep 18 14:37:16.209524 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:16.209485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg" event={"ID":"2413cf32-f7ce-4f80-8529-e54027a20a66","Type":"ContainerStarted","Data":"5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494"}
Sep 18 14:37:16.209895 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:16.209538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg" event={"ID":"2413cf32-f7ce-4f80-8529-e54027a20a66","Type":"ContainerStarted","Data":"5fe3d2b81618b677129de8623854983affd82acd16c14507d614a78bad60966b"}
Sep 18 14:37:16.260467 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:16.260421    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg" podStartSLOduration=2.2604068440000002 podStartE2EDuration="2.260406844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:37:16.258776978 +0000 UTC m=+2324.415297257" watchObservedRunningTime="2026-09-18 14:37:16.260406844 +0000 UTC m=+2324.416927162"
Sep 18 14:37:16.960636 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:16.960598    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:37:25.290757 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:25.290672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:25.290757 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:25.290723    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:25.302635 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:25.302609    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:26.278788 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:26.278761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:37:48.163352 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:48.163315    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"]
Sep 18 14:37:48.938049 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:48.938009    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"]
Sep 18 14:37:51.318282 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:37:51.318238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:38:01.187021 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:01.186977    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:38:03.182548 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.182471    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps" podUID="6e779c16-c8ed-4229-99c0-4202f7684cb0" containerName="main" containerID="cri-o://2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4" gracePeriod=45
Sep 18 14:38:03.434969 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.434913    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:38:03.480580 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.480553    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e779c16-c8ed-4229-99c0-4202f7684cb0" containerID="2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4" exitCode=0
Sep 18 14:38:03.480712 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.480597    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"
Sep 18 14:38:03.480712 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.480641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps" event={"ID":"6e779c16-c8ed-4229-99c0-4202f7684cb0","Type":"ContainerDied","Data":"2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4"}
Sep 18 14:38:03.480712 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.480685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps" event={"ID":"6e779c16-c8ed-4229-99c0-4202f7684cb0","Type":"ContainerDied","Data":"d1150a80a5e2bcd4fd6dd0016021dce9d59e60dbbcaf8a78466dd5d02f9cc438"}
Sep 18 14:38:03.480712 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.480704    2563 scope.go:117] "RemoveContainer" containerID="2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4"
Sep 18 14:38:03.488854 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.488835    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-dshm\") pod \"6e779c16-c8ed-4229-99c0-4202f7684cb0\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") "
Sep 18 14:38:03.488948 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.488870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mnl8\" (UniqueName: \"kubernetes.io/projected/6e779c16-c8ed-4229-99c0-4202f7684cb0-kube-api-access-4mnl8\") pod \"6e779c16-c8ed-4229-99c0-4202f7684cb0\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") "
Sep 18 14:38:03.488948 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.488888    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-tmp-dir\") pod \"6e779c16-c8ed-4229-99c0-4202f7684cb0\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") "
Sep 18 14:38:03.489082 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.489063    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-model-cache\") pod \"6e779c16-c8ed-4229-99c0-4202f7684cb0\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") "
Sep 18 14:38:03.489376 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.489357    2563 scope.go:117] "RemoveContainer" containerID="2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4"
Sep 18 14:38:03.489453 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.489168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e779c16-c8ed-4229-99c0-4202f7684cb0" (UID: "6e779c16-c8ed-4229-99c0-4202f7684cb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:38:03.489519 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.489483    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e779c16-c8ed-4229-99c0-4202f7684cb0" (UID: "6e779c16-c8ed-4229-99c0-4202f7684cb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:38:03.489608 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.489585    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e779c16-c8ed-4229-99c0-4202f7684cb0-tls-certs\") pod \"6e779c16-c8ed-4229-99c0-4202f7684cb0\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") "
Sep 18 14:38:03.489669 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.489627    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-home\") pod \"6e779c16-c8ed-4229-99c0-4202f7684cb0\" (UID: \"6e779c16-c8ed-4229-99c0-4202f7684cb0\") "
Sep 18 14:38:03.489729 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:38:03.489660    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4\": container with ID starting with 2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4 not found: ID does not exist" containerID="2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4"
Sep 18 14:38:03.489729 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.489693    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4"} err="failed to get container status \"2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4\": rpc error: code = NotFound desc = could not find container \"2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4\": container with ID starting with 2a647893eb552c3b55510cd6e29c115ad9361ff876ff73943da42d39f44b2ab4 not found: ID does not exist"
Sep 18 14:38:03.489885 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.489863    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-home" (OuterVolumeSpecName: "home") pod "6e779c16-c8ed-4229-99c0-4202f7684cb0" (UID: "6e779c16-c8ed-4229-99c0-4202f7684cb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:38:03.489885 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.489874    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:03.489969 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.489897    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:03.491158 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.491115    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-dshm" (OuterVolumeSpecName: "dshm") pod "6e779c16-c8ed-4229-99c0-4202f7684cb0" (UID: "6e779c16-c8ed-4229-99c0-4202f7684cb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:38:03.491249 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.491188    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e779c16-c8ed-4229-99c0-4202f7684cb0-kube-api-access-4mnl8" (OuterVolumeSpecName: "kube-api-access-4mnl8") pod "6e779c16-c8ed-4229-99c0-4202f7684cb0" (UID: "6e779c16-c8ed-4229-99c0-4202f7684cb0"). InnerVolumeSpecName "kube-api-access-4mnl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:38:03.491488 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.491467    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e779c16-c8ed-4229-99c0-4202f7684cb0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e779c16-c8ed-4229-99c0-4202f7684cb0" (UID: "6e779c16-c8ed-4229-99c0-4202f7684cb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:38:03.590374 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.590352    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mnl8\" (UniqueName: \"kubernetes.io/projected/6e779c16-c8ed-4229-99c0-4202f7684cb0-kube-api-access-4mnl8\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:03.590374 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.590373    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e779c16-c8ed-4229-99c0-4202f7684cb0-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:03.590502 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.590382    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:03.590502 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.590389    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e779c16-c8ed-4229-99c0-4202f7684cb0-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:03.812597 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.812567    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"]
Sep 18 14:38:03.824742 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.824715    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-355c1f75-kserve-77d4c5c5f-vkcps"]
Sep 18 14:38:03.956650 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:03.956615    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg" podUID="2413cf32-f7ce-4f80-8529-e54027a20a66" containerName="main" containerID="cri-o://5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494" gracePeriod=45
Sep 18 14:38:04.187828 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.187800    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:38:04.287375 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.287350    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e779c16-c8ed-4229-99c0-4202f7684cb0" path="/var/lib/kubelet/pods/6e779c16-c8ed-4229-99c0-4202f7684cb0/volumes"
Sep 18 14:38:04.294989 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.294970    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-home\") pod \"2413cf32-f7ce-4f80-8529-e54027a20a66\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") "
Sep 18 14:38:04.295096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.295010    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdfvz\" (UniqueName: \"kubernetes.io/projected/2413cf32-f7ce-4f80-8529-e54027a20a66-kube-api-access-mdfvz\") pod \"2413cf32-f7ce-4f80-8529-e54027a20a66\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") "
Sep 18 14:38:04.295096 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.295059    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2413cf32-f7ce-4f80-8529-e54027a20a66-tls-certs\") pod \"2413cf32-f7ce-4f80-8529-e54027a20a66\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") "
Sep 18 14:38:04.295224 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.295108    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-dshm\") pod \"2413cf32-f7ce-4f80-8529-e54027a20a66\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") "
Sep 18 14:38:04.295224 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.295204    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-home" (OuterVolumeSpecName: "home") pod "2413cf32-f7ce-4f80-8529-e54027a20a66" (UID: "2413cf32-f7ce-4f80-8529-e54027a20a66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:38:04.295323 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.295270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-model-cache\") pod \"2413cf32-f7ce-4f80-8529-e54027a20a66\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") "
Sep 18 14:38:04.295323 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.295318    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-tmp-dir\") pod \"2413cf32-f7ce-4f80-8529-e54027a20a66\" (UID: \"2413cf32-f7ce-4f80-8529-e54027a20a66\") "
Sep 18 14:38:04.295526 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.295504    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2413cf32-f7ce-4f80-8529-e54027a20a66" (UID: "2413cf32-f7ce-4f80-8529-e54027a20a66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:38:04.295526 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.295519    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-home\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:04.295526 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.295521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-model-cache" (OuterVolumeSpecName: "model-cache") pod "2413cf32-f7ce-4f80-8529-e54027a20a66" (UID: "2413cf32-f7ce-4f80-8529-e54027a20a66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:38:04.297085 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.297066    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2413cf32-f7ce-4f80-8529-e54027a20a66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2413cf32-f7ce-4f80-8529-e54027a20a66" (UID: "2413cf32-f7ce-4f80-8529-e54027a20a66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:38:04.297547 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.297518    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-dshm" (OuterVolumeSpecName: "dshm") pod "2413cf32-f7ce-4f80-8529-e54027a20a66" (UID: "2413cf32-f7ce-4f80-8529-e54027a20a66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 14:38:04.297547 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.297528    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2413cf32-f7ce-4f80-8529-e54027a20a66-kube-api-access-mdfvz" (OuterVolumeSpecName: "kube-api-access-mdfvz") pod "2413cf32-f7ce-4f80-8529-e54027a20a66" (UID: "2413cf32-f7ce-4f80-8529-e54027a20a66"). InnerVolumeSpecName "kube-api-access-mdfvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:38:04.396316 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.396247    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2413cf32-f7ce-4f80-8529-e54027a20a66-tls-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:04.396316 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.396276    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-dshm\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:04.396316 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.396289    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-model-cache\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:04.396316 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.396301    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2413cf32-f7ce-4f80-8529-e54027a20a66-tmp-dir\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:04.396316 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.396315    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdfvz\" (UniqueName: \"kubernetes.io/projected/2413cf32-f7ce-4f80-8529-e54027a20a66-kube-api-access-mdfvz\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:04.487653 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.487624    2563 generic.go:356] "Generic (PLEG): container finished" podID="2413cf32-f7ce-4f80-8529-e54027a20a66" containerID="5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494" exitCode=0
Sep 18 14:38:04.487753 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.487665    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"
Sep 18 14:38:04.487753 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.487716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg" event={"ID":"2413cf32-f7ce-4f80-8529-e54027a20a66","Type":"ContainerDied","Data":"5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494"}
Sep 18 14:38:04.487856 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.487758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg" event={"ID":"2413cf32-f7ce-4f80-8529-e54027a20a66","Type":"ContainerDied","Data":"5fe3d2b81618b677129de8623854983affd82acd16c14507d614a78bad60966b"}
Sep 18 14:38:04.487856 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.487781    2563 scope.go:117] "RemoveContainer" containerID="5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494"
Sep 18 14:38:04.497372 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.497352    2563 scope.go:117] "RemoveContainer" containerID="5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494"
Sep 18 14:38:04.497627 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:38:04.497607    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494\": container with ID starting with 5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494 not found: ID does not exist" containerID="5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494"
Sep 18 14:38:04.497689 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.497634    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494"} err="failed to get container status \"5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494\": rpc error: code = NotFound desc = could not find container \"5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494\": container with ID starting with 5765aab0210c1a0051b2018166f5b8fc56a65ec8ef3323168b80698cd38e8494 not found: ID does not exist"
Sep 18 14:38:04.518990 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.518969    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"]
Sep 18 14:38:04.524789 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:04.524768    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-355c1f75-kserve-7bd5864cc9-gh7jg"]
Sep 18 14:38:06.287964 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:06.287918    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2413cf32-f7ce-4f80-8529-e54027a20a66" path="/var/lib/kubelet/pods/2413cf32-f7ce-4f80-8529-e54027a20a66/volumes"
Sep 18 14:38:07.865480 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:07.865434    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:38:08.159567 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.159491    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"]
Sep 18 14:38:08.159870 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.159852    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2413cf32-f7ce-4f80-8529-e54027a20a66" containerName="main"
Sep 18 14:38:08.159915 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.159873    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2413cf32-f7ce-4f80-8529-e54027a20a66" containerName="main"
Sep 18 14:38:08.159915 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.159894    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e779c16-c8ed-4229-99c0-4202f7684cb0" containerName="main"
Sep 18 14:38:08.159915 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.159900    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e779c16-c8ed-4229-99c0-4202f7684cb0" containerName="main"
Sep 18 14:38:08.160021 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.159962    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e779c16-c8ed-4229-99c0-4202f7684cb0" containerName="main"
Sep 18 14:38:08.160021 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.159972    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2413cf32-f7ce-4f80-8529-e54027a20a66" containerName="main"
Sep 18 14:38:08.164399 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.164377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.189975 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.189948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 14:38:08.190106 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.189957    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 14:38:08.190106 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.189986    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5nh5f\""
Sep 18 14:38:08.191424 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.191407    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 14:38:08.196106 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.196089    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 14:38:08.227236 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.227213    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rns8c\" (UniqueName: \"kubernetes.io/projected/9629ae6f-db57-4fee-9b96-82a855de3e7f-kube-api-access-rns8c\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.227361 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.227242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9629ae6f-db57-4fee-9b96-82a855de3e7f-openshift-service-ca\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.227361 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.227280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-metrics-certs\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.227361 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.227351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-maas-api-tls\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.229016 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.228991    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"]
Sep 18 14:38:08.327747 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.327724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-metrics-certs\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.327846 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.327762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-maas-api-tls\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.327846 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.327797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rns8c\" (UniqueName: \"kubernetes.io/projected/9629ae6f-db57-4fee-9b96-82a855de3e7f-kube-api-access-rns8c\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.327846 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.327812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9629ae6f-db57-4fee-9b96-82a855de3e7f-openshift-service-ca\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.328398 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.328376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9629ae6f-db57-4fee-9b96-82a855de3e7f-openshift-service-ca\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.330097 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.330073    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-maas-api-tls\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.330228 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.330109    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-metrics-certs\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.342035 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.342015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rns8c\" (UniqueName: \"kubernetes.io/projected/9629ae6f-db57-4fee-9b96-82a855de3e7f-kube-api-access-rns8c\") pod \"maas-api-55f6c99747-tmmls\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") " pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.475408 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.475386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:08.759900 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.759872    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"]
Sep 18 14:38:08.761281 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:38:08.761259    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9629ae6f_db57_4fee_9b96_82a855de3e7f.slice/crio-958a3fe2d2f53835c8c6df5077f4f50af826e97c1cf9ca8d4451cadbdfb0ebcb WatchSource:0}: Error finding container 958a3fe2d2f53835c8c6df5077f4f50af826e97c1cf9ca8d4451cadbdfb0ebcb: Status 404 returned error can't find the container with id 958a3fe2d2f53835c8c6df5077f4f50af826e97c1cf9ca8d4451cadbdfb0ebcb
Sep 18 14:38:08.762597 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:08.762577    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 14:38:09.519341 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:09.519309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls" event={"ID":"9629ae6f-db57-4fee-9b96-82a855de3e7f","Type":"ContainerStarted","Data":"8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53"}
Sep 18 14:38:09.519701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:09.519362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls" event={"ID":"9629ae6f-db57-4fee-9b96-82a855de3e7f","Type":"ContainerStarted","Data":"958a3fe2d2f53835c8c6df5077f4f50af826e97c1cf9ca8d4451cadbdfb0ebcb"}
Sep 18 14:38:09.519701 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:09.519390    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:09.552825 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:09.552776    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls" podStartSLOduration=0.95622109 podStartE2EDuration="1.55276111s" totalImagesPullingTime="596.54002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:38:09.548174622 +0000 UTC m=+2377.704694899" watchObservedRunningTime="2026-09-18 14:38:09.55276111 +0000 UTC m=+2377.709281390"
Sep 18 14:38:15.530952 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:15.530923    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:16.232604 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.232567    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"]
Sep 18 14:38:16.235980 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.235963    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.241460 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.241436    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:38:16.264462 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.264442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"]
Sep 18 14:38:16.392983 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.392956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjkkk\" (UniqueName: \"kubernetes.io/projected/e87f2ff7-a211-495f-ae26-1b6de3103802-kube-api-access-zjkkk\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.393111 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.393069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87f2ff7-a211-495f-ae26-1b6de3103802-metrics-certs\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.393217 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.393108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87f2ff7-a211-495f-ae26-1b6de3103802-openshift-service-ca\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.393338 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.393311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87f2ff7-a211-495f-ae26-1b6de3103802-maas-api-tls\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.494357 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.494302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87f2ff7-a211-495f-ae26-1b6de3103802-openshift-service-ca\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.494447 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.494355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87f2ff7-a211-495f-ae26-1b6de3103802-maas-api-tls\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.494447 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.494390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkkk\" (UniqueName: \"kubernetes.io/projected/e87f2ff7-a211-495f-ae26-1b6de3103802-kube-api-access-zjkkk\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.494560 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.494450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87f2ff7-a211-495f-ae26-1b6de3103802-metrics-certs\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.494889 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.494869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e87f2ff7-a211-495f-ae26-1b6de3103802-openshift-service-ca\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.497099 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.497080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e87f2ff7-a211-495f-ae26-1b6de3103802-metrics-certs\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.497258 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.497238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e87f2ff7-a211-495f-ae26-1b6de3103802-maas-api-tls\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.504487 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.504465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkkk\" (UniqueName: \"kubernetes.io/projected/e87f2ff7-a211-495f-ae26-1b6de3103802-kube-api-access-zjkkk\") pod \"maas-api-697bd64575-2k2gf\" (UID: \"e87f2ff7-a211-495f-ae26-1b6de3103802\") " pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.546224 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.546206    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:16.841432 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:16.841405    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"]
Sep 18 14:38:16.843120 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:38:16.843090    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87f2ff7_a211_495f_ae26_1b6de3103802.slice/crio-e7049e8704a648eb22281e0a63409bab3d6f49f63a5879033069e0e745509e98 WatchSource:0}: Error finding container e7049e8704a648eb22281e0a63409bab3d6f49f63a5879033069e0e745509e98: Status 404 returned error can't find the container with id e7049e8704a648eb22281e0a63409bab3d6f49f63a5879033069e0e745509e98
Sep 18 14:38:17.567464 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:17.567391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf" event={"ID":"e87f2ff7-a211-495f-ae26-1b6de3103802","Type":"ContainerStarted","Data":"6ec39a7f39b7cf36f49fe549e4b3f860d935a413b35f4457e453ecadb01d9d50"}
Sep 18 14:38:17.567464 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:17.567448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf" event={"ID":"e87f2ff7-a211-495f-ae26-1b6de3103802","Type":"ContainerStarted","Data":"e7049e8704a648eb22281e0a63409bab3d6f49f63a5879033069e0e745509e98"}
Sep 18 14:38:17.567825 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:17.567470    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:17.604167 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:17.604106    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf" podStartSLOduration=1.248663586 podStartE2EDuration="1.604090339s" totalImagesPullingTime="355.426753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:38:17.596325684 +0000 UTC m=+2385.752845964" watchObservedRunningTime="2026-09-18 14:38:17.604090339 +0000 UTC m=+2385.760610618"
Sep 18 14:38:22.951680 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:22.951638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:38:23.578322 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:23.578292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-697bd64575-2k2gf"
Sep 18 14:38:23.659198 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:23.659163    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"]
Sep 18 14:38:23.659443 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:23.659396    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls" podUID="9629ae6f-db57-4fee-9b96-82a855de3e7f" containerName="maas-api" containerID="cri-o://8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53" gracePeriod=30
Sep 18 14:38:24.901669 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:24.901645    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:25.062397 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.062332    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rns8c\" (UniqueName: \"kubernetes.io/projected/9629ae6f-db57-4fee-9b96-82a855de3e7f-kube-api-access-rns8c\") pod \"9629ae6f-db57-4fee-9b96-82a855de3e7f\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") "
Sep 18 14:38:25.062397 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.062396    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-maas-api-tls\") pod \"9629ae6f-db57-4fee-9b96-82a855de3e7f\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") "
Sep 18 14:38:25.062548 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.062425    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-metrics-certs\") pod \"9629ae6f-db57-4fee-9b96-82a855de3e7f\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") "
Sep 18 14:38:25.062548 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.062447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9629ae6f-db57-4fee-9b96-82a855de3e7f-openshift-service-ca\") pod \"9629ae6f-db57-4fee-9b96-82a855de3e7f\" (UID: \"9629ae6f-db57-4fee-9b96-82a855de3e7f\") "
Sep 18 14:38:25.062862 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.062822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9629ae6f-db57-4fee-9b96-82a855de3e7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9629ae6f-db57-4fee-9b96-82a855de3e7f" (UID: "9629ae6f-db57-4fee-9b96-82a855de3e7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 14:38:25.064396 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.064374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9629ae6f-db57-4fee-9b96-82a855de3e7f-kube-api-access-rns8c" (OuterVolumeSpecName: "kube-api-access-rns8c") pod "9629ae6f-db57-4fee-9b96-82a855de3e7f" (UID: "9629ae6f-db57-4fee-9b96-82a855de3e7f"). InnerVolumeSpecName "kube-api-access-rns8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 14:38:25.064522 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.064414    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9629ae6f-db57-4fee-9b96-82a855de3e7f" (UID: "9629ae6f-db57-4fee-9b96-82a855de3e7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:38:25.064522 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.064467    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9629ae6f-db57-4fee-9b96-82a855de3e7f" (UID: "9629ae6f-db57-4fee-9b96-82a855de3e7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 14:38:25.163176 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.163149    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-maas-api-tls\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:25.163176 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.163175    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9629ae6f-db57-4fee-9b96-82a855de3e7f-metrics-certs\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:25.163176 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.163184    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9629ae6f-db57-4fee-9b96-82a855de3e7f-openshift-service-ca\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:25.163329 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.163193    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rns8c\" (UniqueName: \"kubernetes.io/projected/9629ae6f-db57-4fee-9b96-82a855de3e7f-kube-api-access-rns8c\") on node \"ip-10-0-139-55.ec2.internal\" DevicePath \"\""
Sep 18 14:38:25.611629 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.611598    2563 generic.go:356] "Generic (PLEG): container finished" podID="9629ae6f-db57-4fee-9b96-82a855de3e7f" containerID="8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53" exitCode=0
Sep 18 14:38:25.611724 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.611639    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"
Sep 18 14:38:25.611724 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.611685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls" event={"ID":"9629ae6f-db57-4fee-9b96-82a855de3e7f","Type":"ContainerDied","Data":"8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53"}
Sep 18 14:38:25.611797 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.611733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55f6c99747-tmmls" event={"ID":"9629ae6f-db57-4fee-9b96-82a855de3e7f","Type":"ContainerDied","Data":"958a3fe2d2f53835c8c6df5077f4f50af826e97c1cf9ca8d4451cadbdfb0ebcb"}
Sep 18 14:38:25.611797 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.611757    2563 scope.go:117] "RemoveContainer" containerID="8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53"
Sep 18 14:38:25.621184 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.621167    2563 scope.go:117] "RemoveContainer" containerID="8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53"
Sep 18 14:38:25.621428 ip-10-0-139-55 kubenswrapper[2563]: E0918 14:38:25.621410    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53\": container with ID starting with 8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53 not found: ID does not exist" containerID="8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53"
Sep 18 14:38:25.621474 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.621446    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53"} err="failed to get container status \"8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53\": rpc error: code = NotFound desc = could not find container \"8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53\": container with ID starting with 8e9ca7c398a95e3937749b980a509ec5456a0a28bb72d12823580734bfd78d53 not found: ID does not exist"
Sep 18 14:38:25.636078 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.636054    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"]
Sep 18 14:38:25.640980 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:25.640959    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55f6c99747-tmmls"]
Sep 18 14:38:26.287994 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:26.287968    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9629ae6f-db57-4fee-9b96-82a855de3e7f" path="/var/lib/kubelet/pods/9629ae6f-db57-4fee-9b96-82a855de3e7f/volumes"
Sep 18 14:38:28.829246 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:28.829209    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:38:32.910787 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:32.910761    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:38:32.928732 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:32.928712    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:38:35.958882 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:35.958831    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-ptvsj"]
Sep 18 14:38:45.976061 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:45.976025    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d96fdcf9-x4ffv_bcbfc919-f258-4ace-ba7c-bce32f56f64d/copy-manifests/0.log"
Sep 18 14:38:45.996367 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:45.996335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d96fdcf9-x4ffv_bcbfc919-f258-4ace-ba7c-bce32f56f64d/manager/0.log"
Sep 18 14:38:46.333400 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:46.333327    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-697bd64575-2k2gf_e87f2ff7-a211-495f-ae26-1b6de3103802/maas-api/0.log"
Sep 18 14:38:46.561630 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:46.561604    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829015-jthml_725a8a0f-50f4-443d-b874-fe929e24093e/cleanup/0.log"
Sep 18 14:38:47.913946 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:47.913919    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-hnssn_14aa193a-7901-4581-9ba3-3b110c268113/kuadrant-console-plugin/0.log"
Sep 18 14:38:48.158760 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:48.158726    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-ptvsj_853bb56f-f351-4065-a812-bac5160b5e34/limitador/0.log"
Sep 18 14:38:48.925345 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:48.925309    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc75d4b5-bzxf7_586bc67f-67ae-4daf-a65d-4791833eae78/kube-auth-proxy/0.log"
Sep 18 14:38:49.574906 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:49.574873    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-764b645cd6-x9x9r_2bc70927-d434-49f3-b09a-dda6af701e30/router/0.log"
Sep 18 14:38:50.422151 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:50.422106    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn8xxf_88eea5fd-f516-4385-aba3-e7d934283539/main/0.log"
Sep 18 14:38:50.691762 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:50.691694    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-zcl8r_ab3de34a-f278-452a-82ba-6d9c4224670d/main/0.log"
Sep 18 14:38:56.901501 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:56.901465    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xtsh/must-gather-pwvxr"]
Sep 18 14:38:56.903999 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:56.901827    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9629ae6f-db57-4fee-9b96-82a855de3e7f" containerName="maas-api"
Sep 18 14:38:56.903999 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:56.901841    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9629ae6f-db57-4fee-9b96-82a855de3e7f" containerName="maas-api"
Sep 18 14:38:56.903999 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:56.901914    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9629ae6f-db57-4fee-9b96-82a855de3e7f" containerName="maas-api"
Sep 18 14:38:56.904953 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:56.904935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xtsh/must-gather-pwvxr"
Sep 18 14:38:56.912457 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:56.912436    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xtsh\"/\"openshift-service-ca.crt\""
Sep 18 14:38:56.912956 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:56.912936    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xtsh\"/\"kube-root-ca.crt\""
Sep 18 14:38:56.913409 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:56.913391    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9xtsh\"/\"default-dockercfg-r9lr6\""
Sep 18 14:38:56.949072 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:56.949041    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xtsh/must-gather-pwvxr"]
Sep 18 14:38:57.002041 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:57.002016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5539440b-b614-4ebb-ab35-b5c7a9b3d834-must-gather-output\") pod \"must-gather-pwvxr\" (UID: \"5539440b-b614-4ebb-ab35-b5c7a9b3d834\") " pod="openshift-must-gather-9xtsh/must-gather-pwvxr"
Sep 18 14:38:57.002148 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:57.002051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97hq5\" (UniqueName: \"kubernetes.io/projected/5539440b-b614-4ebb-ab35-b5c7a9b3d834-kube-api-access-97hq5\") pod \"must-gather-pwvxr\" (UID: \"5539440b-b614-4ebb-ab35-b5c7a9b3d834\") " pod="openshift-must-gather-9xtsh/must-gather-pwvxr"
Sep 18 14:38:57.103163 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:57.103113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5539440b-b614-4ebb-ab35-b5c7a9b3d834-must-gather-output\") pod \"must-gather-pwvxr\" (UID: \"5539440b-b614-4ebb-ab35-b5c7a9b3d834\") " pod="openshift-must-gather-9xtsh/must-gather-pwvxr"
Sep 18 14:38:57.103254 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:57.103176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97hq5\" (UniqueName: \"kubernetes.io/projected/5539440b-b614-4ebb-ab35-b5c7a9b3d834-kube-api-access-97hq5\") pod \"must-gather-pwvxr\" (UID: \"5539440b-b614-4ebb-ab35-b5c7a9b3d834\") " pod="openshift-must-gather-9xtsh/must-gather-pwvxr"
Sep 18 14:38:57.103425 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:57.103406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5539440b-b614-4ebb-ab35-b5c7a9b3d834-must-gather-output\") pod \"must-gather-pwvxr\" (UID: \"5539440b-b614-4ebb-ab35-b5c7a9b3d834\") " pod="openshift-must-gather-9xtsh/must-gather-pwvxr"
Sep 18 14:38:57.114545 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:57.114523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97hq5\" (UniqueName: \"kubernetes.io/projected/5539440b-b614-4ebb-ab35-b5c7a9b3d834-kube-api-access-97hq5\") pod \"must-gather-pwvxr\" (UID: \"5539440b-b614-4ebb-ab35-b5c7a9b3d834\") " pod="openshift-must-gather-9xtsh/must-gather-pwvxr"
Sep 18 14:38:57.214719 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:57.214698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xtsh/must-gather-pwvxr"
Sep 18 14:38:57.500148 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:57.500094    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xtsh/must-gather-pwvxr"]
Sep 18 14:38:57.502734 ip-10-0-139-55 kubenswrapper[2563]: W0918 14:38:57.502705    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5539440b_b614_4ebb_ab35_b5c7a9b3d834.slice/crio-ba134b0612f4bf18fd38430b54a209eaf559b9000e466b73ce7ce90aa4b835d5 WatchSource:0}: Error finding container ba134b0612f4bf18fd38430b54a209eaf559b9000e466b73ce7ce90aa4b835d5: Status 404 returned error can't find the container with id ba134b0612f4bf18fd38430b54a209eaf559b9000e466b73ce7ce90aa4b835d5
Sep 18 14:38:57.800101 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:57.800039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xtsh/must-gather-pwvxr" event={"ID":"5539440b-b614-4ebb-ab35-b5c7a9b3d834","Type":"ContainerStarted","Data":"ba134b0612f4bf18fd38430b54a209eaf559b9000e466b73ce7ce90aa4b835d5"}
Sep 18 14:38:58.808195 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:58.808158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xtsh/must-gather-pwvxr" event={"ID":"5539440b-b614-4ebb-ab35-b5c7a9b3d834","Type":"ContainerStarted","Data":"d11e06532a64030effa4b77fd16b8baf6737c93d3a76dd67786d90e00bd90fdf"}
Sep 18 14:38:58.808633 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:58.808211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xtsh/must-gather-pwvxr" event={"ID":"5539440b-b614-4ebb-ab35-b5c7a9b3d834","Type":"ContainerStarted","Data":"c9d361f2f314af1bf37000e1ba5de35781982d63ec0dc947b5b379d6356a5778"}
Sep 18 14:38:58.835858 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:38:58.835806    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9xtsh/must-gather-pwvxr" podStartSLOduration=1.9955438170000002 podStartE2EDuration="2.835790827s" totalImagesPullingTime="840.24701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 14:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:38:58.832459262 +0000 UTC m=+2426.988979547" watchObservedRunningTime="2026-09-18 14:38:58.835790827 +0000 UTC m=+2426.992311124"
Sep 18 14:39:00.250554 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:00.250515    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pp5sf_6812878a-b037-40b6-89c1-445d49376f65/global-pull-secret-syncer/0.log"
Sep 18 14:39:00.492574 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:00.492542    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wtgbx_17faa55e-6e23-473c-828c-8727a86c9977/konnectivity-agent/0.log"
Sep 18 14:39:00.606032 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:00.605952    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-55.ec2.internal_3e878fac6c20a59b3277411748de820b/haproxy/0.log"
Sep 18 14:39:04.960897 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:04.960853    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-hnssn_14aa193a-7901-4581-9ba3-3b110c268113/kuadrant-console-plugin/0.log"
Sep 18 14:39:05.025898 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:05.025870    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-ptvsj_853bb56f-f351-4065-a812-bac5160b5e34/limitador/0.log"
Sep 18 14:39:06.888730 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:06.888694    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-fc4t7_1e8a3b04-7240-4c24-a4c2-133a57d91595/cluster-monitoring-operator/0.log"
Sep 18 14:39:07.511766 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:07.511738    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dt75n_08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9/node-exporter/0.log"
Sep 18 14:39:07.576783 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:07.576750    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dt75n_08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9/kube-rbac-proxy/0.log"
Sep 18 14:39:07.654006 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:07.653930    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dt75n_08c9cfc7-d31b-4ebd-9a96-3ec8f3d54ec9/init-textfile/0.log"
Sep 18 14:39:08.860845 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:08.860820    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-r5l42_6782b4e4-dfd5-43b9-ab12-d3c049f0c930/prometheus-operator-admission-webhook/0.log"
Sep 18 14:39:09.860993 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:09.860953    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"]
Sep 18 14:39:09.866574 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:09.866543    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:09.891071 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:09.891037    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"]
Sep 18 14:39:09.933959 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:09.933920    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-proc\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:09.934180 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:09.934026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-lib-modules\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:09.934180 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:09.934062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdtgs\" (UniqueName: \"kubernetes.io/projected/16d2c039-3243-4959-85b7-4fc171c1c5bb-kube-api-access-rdtgs\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:09.934180 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:09.934080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-podres\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:09.934180 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:09.934105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-sys\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.035735 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.035694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-proc\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.035934 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.035791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-lib-modules\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.035934 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.035826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-proc\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.035934 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.035832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdtgs\" (UniqueName: \"kubernetes.io/projected/16d2c039-3243-4959-85b7-4fc171c1c5bb-kube-api-access-rdtgs\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.035934 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.035914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-podres\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.036189 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.035953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-lib-modules\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.036189 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.035959    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-sys\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.036189 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.036018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-podres\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.036189 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.036104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16d2c039-3243-4959-85b7-4fc171c1c5bb-sys\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.045836 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.045814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdtgs\" (UniqueName: \"kubernetes.io/projected/16d2c039-3243-4959-85b7-4fc171c1c5bb-kube-api-access-rdtgs\") pod \"perf-node-gather-daemonset-jvk9w\" (UID: \"16d2c039-3243-4959-85b7-4fc171c1c5bb\") " pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.178564 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.178527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.483098 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.483070    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-lpn5x_ff7fb1d3-fbc3-4397-afe8-fa68ae822717/networking-console-plugin/0.log"
Sep 18 14:39:10.484402 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.484374    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"]
Sep 18 14:39:10.906565 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.906488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w" event={"ID":"16d2c039-3243-4959-85b7-4fc171c1c5bb","Type":"ContainerStarted","Data":"d002f40443d8b885a3d8f8470437eec26f2a0d7c3e2bf56b69abbd53ad1b4ea5"}
Sep 18 14:39:10.906565 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.906542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w" event={"ID":"16d2c039-3243-4959-85b7-4fc171c1c5bb","Type":"ContainerStarted","Data":"35c9551957ccd9d686f4313e55b5619753cf3873c84620536d2fcdb4a49a5ef2"}
Sep 18 14:39:10.906947 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.906574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:10.944063 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:10.944001    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w" podStartSLOduration=1.943980792 podStartE2EDuration="1.943980792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 14:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 14:39:10.936439408 +0000 UTC m=+2439.092959689" watchObservedRunningTime="2026-09-18 14:39:10.943980792 +0000 UTC m=+2439.100501073"
Sep 18 14:39:11.138685 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:11.138652    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/2.log"
Sep 18 14:39:11.142715 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:11.142690    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-v2rtv_d72a0a56-0157-4b87-8ddf-34754f6bdaa3/console-operator/3.log"
Sep 18 14:39:11.787357 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:11.787329    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6499b5bf89-8gs7t_ef400e89-77bb-4aed-8c10-090cbd822016/console/0.log"
Sep 18 14:39:11.875495 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:11.875448    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-nn5ds_23f2129e-d074-425b-aa31-7b53bd82749d/download-server/0.log"
Sep 18 14:39:12.619170 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:12.619143    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-9wlm8_2e656651-372f-4cc7-9778-ba467cb45310/volume-data-source-validator/0.log"
Sep 18 14:39:14.110049 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:14.110017    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k9kwv_66650627-a6b5-4d1f-b5c6-3f812de684eb/dns/0.log"
Sep 18 14:39:14.177236 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:14.177211    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k9kwv_66650627-a6b5-4d1f-b5c6-3f812de684eb/kube-rbac-proxy/0.log"
Sep 18 14:39:14.287318 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:14.287285    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-plgx7_94143e9a-35af-4e2f-92fe-e27c520db0ab/dns-node-resolver/0.log"
Sep 18 14:39:15.119712 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:15.119657    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-98pdk_336c2a92-68fa-4de6-ba21-fbeab8836c37/node-ca/0.log"
Sep 18 14:39:16.923708 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:16.923679    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9xtsh/perf-node-gather-daemonset-jvk9w"
Sep 18 14:39:16.993297 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:16.993267    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc75d4b5-bzxf7_586bc67f-67ae-4daf-a65d-4791833eae78/kube-auth-proxy/0.log"
Sep 18 14:39:17.492151 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:17.492099    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-764b645cd6-x9x9r_2bc70927-d434-49f3-b09a-dda6af701e30/router/0.log"
Sep 18 14:39:18.682985 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:18.682963    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bjl8s_d0d7ff07-d797-44c6-8c50-9312493190c1/serve-healthcheck-canary/0.log"
Sep 18 14:39:19.442531 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:19.442302    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-tj226_be1f9fd0-371e-4d59-a868-42ba681905e7/insights-operator/0.log"
Sep 18 14:39:19.445374 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:19.445276    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-tj226_be1f9fd0-371e-4d59-a868-42ba681905e7/insights-operator/1.log"
Sep 18 14:39:19.472634 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:19.472583    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2vl5w_77cd4cd8-dd31-4511-a6ec-1f162d963772/kube-rbac-proxy/0.log"
Sep 18 14:39:19.503054 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:19.503025    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2vl5w_77cd4cd8-dd31-4511-a6ec-1f162d963772/exporter/0.log"
Sep 18 14:39:19.529963 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:19.529937    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2vl5w_77cd4cd8-dd31-4511-a6ec-1f162d963772/extractor/0.log"
Sep 18 14:39:19.969826 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:19.969789    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc75d4b5-bzxf7_586bc67f-67ae-4daf-a65d-4791833eae78/kube-auth-proxy/0.log"
Sep 18 14:39:20.085603 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:20.085521    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-764b645cd6-x9x9r_2bc70927-d434-49f3-b09a-dda6af701e30/router/0.log"
Sep 18 14:39:22.005690 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:22.005657    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc75d4b5-bzxf7_586bc67f-67ae-4daf-a65d-4791833eae78/kube-auth-proxy/0.log"
Sep 18 14:39:22.109226 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:22.109194    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-764b645cd6-x9x9r_2bc70927-d434-49f3-b09a-dda6af701e30/router/0.log"
Sep 18 14:39:22.955189 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:22.955112    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d96fdcf9-x4ffv_bcbfc919-f258-4ace-ba7c-bce32f56f64d/manager/0.log"
Sep 18 14:39:22.973136 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:22.973107    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d96fdcf9-x4ffv_bcbfc919-f258-4ace-ba7c-bce32f56f64d/copy-manifests/0.log"
Sep 18 14:39:24.114746 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:24.114719    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-ff9cccccb-624x7_3af5117d-baa2-4ed3-872b-896519bbc857/manager/0.log"
Sep 18 14:39:24.202064 ip-10-0-139-55 kubenswrapper[2563]: I0918 14:39:24.202037    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-qzhxg_039e20df-12e8-4f1f-a0aa-a4e79c1c759f/openshift-lws-operator/0.log"