Sep 09 20:14:30.408215 ip-10-0-135-122 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:14:30.408229 ip-10-0-135-122 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:14:30.408239 ip-10-0-135-122 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:14:30.408517 ip-10-0-135-122 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:14:40.577659 ip-10-0-135-122 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:14:40.577676 ip-10-0-135-122 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 50c42cac3e314d00a73a9d81d5a49a14 --
Sep 09 20:16:18.382686 ip-10-0-135-122 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:16:18.810406 ip-10-0-135-122 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:16:18.810406 ip-10-0-135-122 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:16:18.810406 ip-10-0-135-122 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:16:18.810406 ip-10-0-135-122 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:16:18.810406 ip-10-0-135-122 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:16:18.811420 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.811299    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:16:18.817523 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817496    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:18.817523 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817515    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:18.817523 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817519    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:18.817523 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817522    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:18.817523 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817528    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:18.817523 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817533    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:18.817523 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817536    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817539    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817542    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817544    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817547    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817550    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817553    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817555    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817558    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817561    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817563    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817574    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817577    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817579    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817582    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817585    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817587    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817590    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817593    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817596    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:18.817783 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817598    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817601    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817603    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817606    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817609    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817611    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817614    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817616    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817619    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817622    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817624    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817626    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817629    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817631    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817635    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817638    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817643    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817645    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817648    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:18.818262 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817651    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817653    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817656    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817660    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817665    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817668    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817672    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817675    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817678    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817681    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817684    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817686    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817689    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817692    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817695    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817697    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817700    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817702    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817705    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:18.818758 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817707    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817710    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817713    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817716    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817718    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817720    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817723    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817727    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817730    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817734    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817738    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817742    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817745    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817748    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817750    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817753    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817756    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817758    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817761    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817764    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:18.819276 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817766    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817769    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817771    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817774    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817777    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817779    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817782    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817785    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817787    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817790    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817793    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817796    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817798    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817801    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.817803    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818556    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818564    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818567    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818570    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818572    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:18.820064 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818576    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818579    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818581    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818584    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818587    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818589    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818593    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818595    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818597    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818600    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818602    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818605    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818608    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818610    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818614    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818616    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818619    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818622    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818624    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818626    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:18.820613 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818629    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818631    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818634    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818636    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818639    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818641    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818644    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818647    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818650    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818652    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818655    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818659    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818664    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818668    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818671    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818674    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818676    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818679    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818682    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:18.821193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818685    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818687    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818689    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818692    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818694    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818697    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818700    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818703    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818705    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818708    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818711    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818713    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818716    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818718    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818720    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818723    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818725    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818728    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818730    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818733    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:18.821691 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818735    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818739    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818743    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818747    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818750    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818753    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818756    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818758    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818761    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818764    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818766    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818771    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818775    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818777    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818781    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818783    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818786    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818789    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818791    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:18.822271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818794    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818796    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818800    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818802    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818805    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818807    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818810    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818812    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818815    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818817    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818821    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818823    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818826    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818828    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818831    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.818833    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821494    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821508    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821512    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821516    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:18.822760 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821518    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821521    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821524    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821527    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821530    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821533    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821535    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821538    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821541    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821543    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821545    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821548    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821551    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821553    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821556    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821558    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821561    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821563    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821566    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821569    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:18.823245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821571    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821574    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821576    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821579    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821581    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821584    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821587    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821590    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821592    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821596    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821599    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821602    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821604    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821607    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821609    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821612    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821614    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821617    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821619    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821622    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:18.823765 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821624    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821629    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821633    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821636    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821639    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821642    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821646    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821649    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821652    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821655    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821658    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821661    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821665    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821668    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821671    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821673    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821676    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821678    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:18.824254 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821682    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821686    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821689    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821693    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821696    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821699    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821701    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821704    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821707    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821709    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821712    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821716    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821718    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821721    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821723    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821726    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821728    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821730    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821733    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821736    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:18.824702 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821738    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821741    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821744    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821747    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821749    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821752    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821754    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821757    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821759    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821762    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821764    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821766    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821769    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821772    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821776    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821779    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.821781    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821867    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821904    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821910    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821915    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:16:18.825198 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821919    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821922    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821926    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821931    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821934    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821937    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821940    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821944    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821947    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821949    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821952    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821955    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821958    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821961    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821966    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821969    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821972    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821975    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821978    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821982    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821985    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821988    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821991    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821994    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:16:18.825782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.821997    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822001    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822004    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822006    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822011    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822014    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822016    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822019    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822022    2581 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822025    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822029    2581 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822032    2581 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822035    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822038    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822041    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822045    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822047    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822051    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822053    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822056    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822060    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822062    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822066    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822069    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822072    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:16:18.826381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822075    2581 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822079    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822081    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822084    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822087    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822090    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822093    2581 flags.go:64] FLAG: --help="false"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822096    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-135-122.ec2.internal"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822099    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822102    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822105    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822108    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822112    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822115    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822117    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822121    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822123    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822126    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822129    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822132    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822135    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822137    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822140    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822143    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:16:18.827015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822145    2581 flags.go:64] FLAG: --lock-file=""
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822148    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822151    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822157    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822163    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822165    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822169    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822172    2581 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822175    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822178    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822181    2581 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822183    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822188    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822191    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822195    2581 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822198    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822201    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822203    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822206    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822209    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822212    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822215    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822223    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822227    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822229    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:16:18.827628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822232    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822235    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822241    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822244    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822248    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822251    2581 flags.go:64] FLAG: --port="10250"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822254    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822256    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-080b9baee7b16d41e"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822260    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822262    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822265    2581 flags.go:64] FLAG: --register-node="true"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822269    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822273    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822276    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822280    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822283    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822286    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822289    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822292    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822295    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822298    2581 flags.go:64] FLAG: --runonce="false"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822301    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822310    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822313    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822316    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:16:18.828253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822318    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822321    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822324    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822328    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822331    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822334    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822336    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822353    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822356    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822359    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822362    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822367    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822370    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822373    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822377    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822380    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822383    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822385    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822389    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822392    2581 flags.go:64] FLAG: --v="2"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822396    2581 flags.go:64] FLAG: --version="false"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822400    2581 flags.go:64] FLAG: --vmodule=""
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822404    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.822408    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822528    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:18.828949 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822532    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822535    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822538    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822541    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822544    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822547    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822549    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822552    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822554    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822557    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822559    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822562    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822564    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822567    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822570    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822574    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822577    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822581    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822584    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:18.829727 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822586    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822589    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822592    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822594    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822597    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822600    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822605    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822608    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822610    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822613    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822615    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822619    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822622    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822625    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822627    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822630    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822632    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822636    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822639    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822641    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:18.830228 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822643    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822646    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822650    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822654    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822657    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822660    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822662    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822665    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822668    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822670    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822673    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822675    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822678    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822680    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822683    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822685    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822688    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822690    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822694    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:18.830737 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822697    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822701    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822704    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822706    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822710    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822713    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822715    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822730    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822732    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822735    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822739    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822741    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822744    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822746    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822749    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822752    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822754    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822757    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822759    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822762    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:18.831193 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822765    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822767    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822770    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822773    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822775    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822778    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822781    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822783    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822785    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822788    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822790    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822794    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822796    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822798    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822801    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822803    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822806    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822808    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822813    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:18.831726 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.822815    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.823478    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.831027    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.831047    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.831938    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.831947    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832012    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832017    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832021    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832023    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832027    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832030    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832034    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832038    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:18.832195 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832041    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832044    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832046    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832048    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832051    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832054    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832057    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832060    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832063    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832082    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832087    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832091    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832094    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832097    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832100    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832103    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832105    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832108    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832112    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:18.832574 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832114    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832117    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832121    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832123    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832126    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832129    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832131    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832134    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832137    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832139    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832142    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832144    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832147    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832149    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832152    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832154    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832157    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832159    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832162    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832166    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:18.833071 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832169    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832171    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832174    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832176    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832179    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832181    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832184    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832186    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832189    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832191    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832194    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832196    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832199    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832201    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832204    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832208    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832210    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832214    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832216    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832219    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:18.833674 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832221    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832224    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832227    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832229    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832232    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832234    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832237    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832239    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832241    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832244    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832246    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832250    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832254    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832257    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832260    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832263    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832266    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832268    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832271    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:18.834171 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832273    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832276    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832279    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832281    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832283    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832286    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832288    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832291    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832294    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832298    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832300    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832303    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832306    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.832311    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832445    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:16:18.834647 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832451    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832454    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832457    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832460    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832462    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832465    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832467    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832470    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832473    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832476    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832479    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832481    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832484    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832487    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832489    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832492    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832494    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832497    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832499    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832502    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832504    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:16:18.835034 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832507    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832510    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832512    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832515    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832518    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832520    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832523    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832526    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832528    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832531    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832533    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832536    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832538    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832540    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832543    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832545    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832548    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832550    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832552    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832555    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:16:18.835603 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832558    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832560    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832563    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832566    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832568    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832571    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832573    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832576    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832579    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832581    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832584    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832587    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832590    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832592    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832595    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832597    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832600    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832603    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832607    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:16:18.836134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832611    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832614    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832616    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832619    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832621    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832624    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832627    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832629    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832632    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832634    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832636    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832639    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832641    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832645    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832649    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832652    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832655    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832657    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832660    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:16:18.836628 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832662    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832665    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832667    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832670    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832672    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832675    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832678    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832680    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832683    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832687    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832690    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832693    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832696    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832698    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832701    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832704    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832707    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832709    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:16:18.837144 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:18.832712    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:16:18.837665 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.832717    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:16:18.837665 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.832849    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:16:18.837665 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.834847    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:16:18.837665 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.835762    2581 server.go:1019] "Starting client certificate rotation"
Sep 09 20:16:18.837665 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.835878    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:16:18.837665 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.836626    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:16:18.838579 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.838563    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:16:18.839797 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.839775    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:16:18.849789 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.849767    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:16:18.852848 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.852830    2581 log.go:25] "Validated CRI v1 image API"
Sep 09 20:16:18.853735 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.853720    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:16:18.858001 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.857980    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7b5b1fd0-25f7-4759-bfa3-736fb0b59fdc:/dev/nvme0n1p3 a5dfa68e-2bb7-4156-84f3-4792e2a63c4e:/dev/nvme0n1p4]
Sep 09 20:16:18.858080 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.857999    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:16:18.863326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.863201    2581 manager.go:216] Machine: {Timestamp:2026-09-09 20:16:18.861923123 +0000 UTC m=+0.355134118 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083746 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2026afa511ae2a7ee98574676232c3 SystemUUID:ec2026af-a511-ae2a-7ee9-8574676232c3 BootID:50c42cac-3e31-4d00-a73a-9d81d5a49a14 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:51:e4:5c:bd:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:51:e4:5c:bd:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:6b:d3:50:af:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:16:18.863326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.863314    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:16:18.863466 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.863450    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:16:18.868204 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.868171    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:16:18.868389 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.868205    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-122.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:16:18.869419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.869397    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:16:18.869419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.869422    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:16:18.869495 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.869434    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:16:18.869713 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.869695    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:16:18.870211 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.870199    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:16:18.871588 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.871577    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:16:18.875352 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.875328    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:16:18.875493 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.875483    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:16:18.875535 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.875515    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:16:18.875535 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.875526    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:16:18.875598 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.875539    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:16:18.875598 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.875553    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:16:18.875598 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.875569    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:16:18.875969 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.875957    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:16:18.876010 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.875976    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:16:18.878377 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.878359    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:16:18.880662 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.880648    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:16:18.880714 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.880680    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:16:18.881622 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881610    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:16:18.881654 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881631    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:16:18.881654 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881638    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:16:18.881654 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881644    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:16:18.881654 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881652    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:16:18.881750 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881660    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:16:18.881750 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881676    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:16:18.881750 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881683    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:16:18.881750 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881694    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:16:18.881750 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881699    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:16:18.881750 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881711    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:16:18.881750 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.881718    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:16:18.882556 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.882546    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:16:18.882556 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.882556    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:16:18.886845 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.886833    2581 server.go:1268] "Started kubelet"
Sep 09 20:16:18.886973 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.886934    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:16:18.887093 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.886932    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:16:18.887093 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.887050    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:16:18.888047 ip-10-0-135-122 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:16:18.888183 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.888165    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:16:18.888459 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.888424    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-122.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:16:18.888550 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.888439    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:16:18.891242 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.891210    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:16:18.893135 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.893114    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:16:18.893945 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.893929    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:16:18.895933 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.895914    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:16:18.896470 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.896456    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:16:18.896576 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.896553    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:18.897089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.897070    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:16:18.897089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.897071    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:16:18.897216 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.897100    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:16:18.897216 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.897188    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:16:18.897216 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.897192    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:16:18.897901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.897881    2581 factory.go:55] Registering systemd factory
Sep 09 20:16:18.897977 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.897905    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:16:18.898260 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.898245    2581 factory.go:153] Registering CRI-O factory
Sep 09 20:16:18.898365 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.898264    2581 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:16:18.898365 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.898321    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:16:18.898365 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.898358    2581 factory.go:103] Registering Raw factory
Sep 09 20:16:18.898525 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.898377    2581 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:16:18.898926 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.898839    2581 manager.go:349] Starting recovery of all containers
Sep 09 20:16:18.900630 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.900596    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:16:18.900724 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.900687    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:16:18.901894 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.900815    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-122.ec2.internal.18d3c05c1f0ec2e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-122.ec2.internal,UID:ip-10-0-135-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-122.ec2.internal,},FirstTimestamp:2026-09-09 20:16:18.8868042 +0000 UTC m=+0.380015174,LastTimestamp:2026-09-09 20:16:18.8868042 +0000 UTC m=+0.380015174,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-122.ec2.internal,}"
Sep 09 20:16:18.906975 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.906956    2581 manager.go:354] Recovery completed
Sep 09 20:16:18.912627 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.912610    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:18.914470 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.914443    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:18.914544 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.914490    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:18.914544 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.914506    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:18.915092 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.915079    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:16:18.915092 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.915092    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:16:18.915176 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.915093    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:18.915176 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.915108    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:16:18.916092 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.916072    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:18.916149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.916114    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:18.916149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.916129    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:18.916604 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.916513    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-122.ec2.internal.18d3c05c20b4ecf2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-122.ec2.internal,UID:ip-10-0-135-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-122.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-122.ec2.internal,},FirstTimestamp:2026-09-09 20:16:18.914471154 +0000 UTC m=+0.407682152,LastTimestamp:2026-09-09 20:16:18.914471154 +0000 UTC m=+0.407682152,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-122.ec2.internal,}"
Sep 09 20:16:18.917281 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.917267    2581 policy_none.go:49] "None policy: Start"
Sep 09 20:16:18.917332 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.917283    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:16:18.917332 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.917293    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:16:18.919412 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.919400    2581 policy_none.go:47] "Start"
Sep 09 20:16:18.926290 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.926222    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-122.ec2.internal.18d3c05c20b5508a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-122.ec2.internal,UID:ip-10-0-135-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-122.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-122.ec2.internal,},FirstTimestamp:2026-09-09 20:16:18.91449665 +0000 UTC m=+0.407707626,LastTimestamp:2026-09-09 20:16:18.91449665 +0000 UTC m=+0.407707626,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-122.ec2.internal,}"
Sep 09 20:16:18.933235 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.933212    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9kndj"
Sep 09 20:16:18.939607 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.939537    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-122.ec2.internal.18d3c05c20b58c04  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-122.ec2.internal,UID:ip-10-0-135-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-122.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-122.ec2.internal,},FirstTimestamp:2026-09-09 20:16:18.914511876 +0000 UTC m=+0.407722860,LastTimestamp:2026-09-09 20:16:18.914511876 +0000 UTC m=+0.407722860,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-122.ec2.internal,}"
Sep 09 20:16:18.940224 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.940199    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9kndj"
Sep 09 20:16:18.983572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.956937    2581 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:16:18.983572 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.956974    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:16:18.983572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.956987    2581 server.go:87] "Starting device plugin registration server"
Sep 09 20:16:18.983572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.957281    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:16:18.983572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.957292    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:16:18.983572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.957408    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:16:18.983572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.957500    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:16:18.983572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.957510    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:16:18.983572 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.958020    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:16:18.983572 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.958066    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:18.994227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.994191    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:16:18.995594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.995574    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:16:18.995688 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.995623    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:16:18.995688 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.995645    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:16:18.995792 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:18.995687    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:16:18.997955 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:18.997937    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:16:19.057993 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.057954    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:19.059021 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.059006    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:19.059107 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.059034    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:19.059107 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.059045    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:19.059107 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.059068    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.070424 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.070368    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.070424 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.070396    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-122.ec2.internal\": node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:19.096404 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.096365    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-122.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal"]
Sep 09 20:16:19.096579 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.096459    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:19.097472 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.097444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f1edbe788da0ad39b612586be095693-config\") pod \"kube-apiserver-proxy-ip-10-0-135-122.ec2.internal\" (UID: \"2f1edbe788da0ad39b612586be095693\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.097472 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.097472    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:19.097642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.097503    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:19.097642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.097514    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:19.099685 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.099672    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:19.099815 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.099799    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.099864 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.099833    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:19.100544 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.100526    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:19.100648 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.100561    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:19.100648 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.100571    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:19.100648 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.100528    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:19.100648 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.100623    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:19.100648 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.100639    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:19.102772 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.102756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.102834 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.102791    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:19.103569 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.103547    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:19.103654 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.103573    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:19.103654 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.103579    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:19.103654 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.103612    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:19.131283 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.131249    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-122.ec2.internal\" not found" node="ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.135774 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.135758    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-122.ec2.internal\" not found" node="ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.197591 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.197553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f1edbe788da0ad39b612586be095693-config\") pod \"kube-apiserver-proxy-ip-10-0-135-122.ec2.internal\" (UID: \"2f1edbe788da0ad39b612586be095693\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.197748 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.197592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/423ca18fda886fc65626bc69f5433aaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal\" (UID: \"423ca18fda886fc65626bc69f5433aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.197748 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.197629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/423ca18fda886fc65626bc69f5433aaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal\" (UID: \"423ca18fda886fc65626bc69f5433aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.197748 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.197695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2f1edbe788da0ad39b612586be095693-config\") pod \"kube-apiserver-proxy-ip-10-0-135-122.ec2.internal\" (UID: \"2f1edbe788da0ad39b612586be095693\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.204015 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.203984    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:19.297839 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.297811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/423ca18fda886fc65626bc69f5433aaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal\" (UID: \"423ca18fda886fc65626bc69f5433aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.297905 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.297843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/423ca18fda886fc65626bc69f5433aaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal\" (UID: \"423ca18fda886fc65626bc69f5433aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.297947 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.297904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/423ca18fda886fc65626bc69f5433aaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal\" (UID: \"423ca18fda886fc65626bc69f5433aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.297947 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.297913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/423ca18fda886fc65626bc69f5433aaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal\" (UID: \"423ca18fda886fc65626bc69f5433aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.304940 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.304912    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:19.405654 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.405578    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:19.433795 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.433768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.439889 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.439316    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:19.506658 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.506602    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:19.607254 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.607218    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:19.707798 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.707717    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:19.808500 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.808453    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:19.835642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.835613    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:16:19.836085 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.835799    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:16:19.909265 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:19.909093    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:19.943643 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.943595    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:11:19 +0000 UTC" deadline="2028-06-22 03:10:55.82976327 +0000 UTC"
Sep 09 20:16:19.943643 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:19.943644    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15630h54m35.886126727s"
Sep 09 20:16:20.010151 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:20.010067    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:20.076296 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.076267    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:16:20.077599 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.077576    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:16:20.077719 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.077610    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:16:20.077719 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.077620    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:16:20.110517 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:20.110477    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:20.211186 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:20.211162    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:20.214758 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.214740    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:16:20.215667 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:20.215636    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1edbe788da0ad39b612586be095693.slice/crio-31dfe1fbc4d01a5b55aa60069ca08738f9bd5f4e33d075759a1c7984b83ac07b WatchSource:0}: Error finding container 31dfe1fbc4d01a5b55aa60069ca08738f9bd5f4e33d075759a1c7984b83ac07b: Status 404 returned error can't find the container with id 31dfe1fbc4d01a5b55aa60069ca08738f9bd5f4e33d075759a1c7984b83ac07b
Sep 09 20:16:20.216271 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:20.216250    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423ca18fda886fc65626bc69f5433aaa.slice/crio-c8d9c309cbc7c3c1313595054a5443ee6a98b3d5062ff9684e454bfe48fba245 WatchSource:0}: Error finding container c8d9c309cbc7c3c1313595054a5443ee6a98b3d5062ff9684e454bfe48fba245: Status 404 returned error can't find the container with id c8d9c309cbc7c3c1313595054a5443ee6a98b3d5062ff9684e454bfe48fba245
Sep 09 20:16:20.222204 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.222184    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:16:20.311789 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:20.311677    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:20.412429 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:20.412390    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-122.ec2.internal\" not found"
Sep 09 20:16:20.425942 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.425917    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:16:20.446664 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.446630    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:16:20.497082 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.497038    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:20.511417 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.511392    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:16:20.512549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.512531    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-122.ec2.internal"
Sep 09 20:16:20.522243 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.522209    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:16:20.876464 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.876428    2581 apiserver.go:52] "Watching apiserver"
Sep 09 20:16:20.889769 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.889729    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:16:20.891029 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.890986    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-n6hd2","openshift-multus/network-metrics-daemon-rbt27","openshift-network-diagnostics/network-check-target-lf5rw","kube-system/konnectivity-agent-sdghr","openshift-image-registry/node-ca-2859f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal","openshift-multus/multus-dm9v2","openshift-network-operator/iptables-alerter-nzgb6","openshift-ovn-kubernetes/ovnkube-node-kpdtc","kube-system/global-pull-secret-syncer-2f9sh","kube-system/kube-apiserver-proxy-ip-10-0-135-122.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97","openshift-cluster-node-tuning-operator/tuned-qdt69","openshift-dns/node-resolver-2xswt"]
Sep 09 20:16:20.894278 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.893744    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.896360 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.896301    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:16:20.896658 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.896637    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:16:20.896909 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.896314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.897025 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.897008    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:16:20.897280 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.897263    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:16:20.897508 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.897476    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:16:20.897749 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.897733    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kg69k\""
Sep 09 20:16:20.898830 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.898810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:20.898927 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:20.898890    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:20.903252 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.903225    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:16:20.903366 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.903315    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:16:20.903591 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.903571    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c9c2n\""
Sep 09 20:16:20.903673 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.903660    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:20.905910 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.905887    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2859f"
Sep 09 20:16:20.907561 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-lib-modules\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.907673 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-var-lib-cni-bin\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.907673 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-kubernetes\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.907673 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-sysctl-conf\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.907673 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-systemd\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.907673 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-cnibin\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.907937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-run-multus-certs\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.907937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-etc-kubernetes\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.907937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-modprobe-d\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.907937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-run\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.907937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-var-lib-kubelet\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.907937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrdcr\" (UniqueName: \"kubernetes.io/projected/b7566bfc-2d5a-4514-b859-3879944bafa7-kube-api-access-hrdcr\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.907937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-sysconfig\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.907937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-sys\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.907937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-var-lib-kubelet\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.907937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-conf-dir\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-sysctl-d\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.907996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-cni-dir\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-run-k8s-cni-cncf-io\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-var-lib-cni-multus\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-tmp\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-run-netns\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-tuned\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlhk5\" (UniqueName: \"kubernetes.io/projected/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-kube-api-access-tlhk5\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4klm\" (UniqueName: \"kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm\") pod \"network-check-target-lf5rw\" (UID: \"9e614f95-303b-473d-b59b-544cc9a5ce53\") " pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-system-cni-dir\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-socket-dir-parent\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-hostroot\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-daemon-config\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.908403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-host\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:20.909102 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2xswt"
Sep 09 20:16:20.909102 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.908451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7566bfc-2d5a-4514-b859-3879944bafa7-cni-binary-copy\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:20.911162 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.911142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:20.913385 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.913354    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:16:20.913525 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.913497    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:16:20.913604 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.913585    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:16:20.913736 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.913709    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:16:20.913839 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.913807    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vf5kz\""
Sep 09 20:16:20.913951 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.913936    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:16:20.914037 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.913959    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qmxpv\""
Sep 09 20:16:20.914096 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.914075    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:16:20.914333 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.914274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:20.914490 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.914343    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:16:20.914490 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.914465    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cd929\""
Sep 09 20:16:20.914782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.914762    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:16:20.914919 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.914794    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:16:20.914919 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.914843    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:16:20.915047 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.914951    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:16:20.915047 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.914951    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j7rzk\""
Sep 09 20:16:20.915258 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.915229    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:16:20.915447 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.915429    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:16:20.916481 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.916462    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:20.916582 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:20.916560    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:20.918363 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.918334    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:16:20.919227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.919201    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:16:20.919416 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.919396    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8sz2c\""
Sep 09 20:16:20.919504 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.919492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:20.919593 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:20.919565    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:20.921324 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.921297    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:16:20.922333 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.922296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nzgb6"
Sep 09 20:16:20.924782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.924762    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:16:20.925128 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.925105    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:20.925233 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.925167    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bbdmt\""
Sep 09 20:16:20.925843 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.925825    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:16:20.926003 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.925986    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:16:20.927849 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.927829    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j84lm\""
Sep 09 20:16:20.927849 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.927840    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:16:20.932430 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.929577    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:16:20.932430 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.929743    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:16:20.948480 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.948457    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lct9x"
Sep 09 20:16:20.957547 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.957474    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lct9x"
Sep 09 20:16:20.998093 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:20.998059    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:16:21.000945 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.000878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-122.ec2.internal" event={"ID":"2f1edbe788da0ad39b612586be095693","Type":"ContainerStarted","Data":"31dfe1fbc4d01a5b55aa60069ca08738f9bd5f4e33d075759a1c7984b83ac07b"}
Sep 09 20:16:21.002457 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.002429    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal" event={"ID":"423ca18fda886fc65626bc69f5433aaa","Type":"ContainerStarted","Data":"c8d9c309cbc7c3c1313595054a5443ee6a98b3d5062ff9684e454bfe48fba245"}
Sep 09 20:16:21.009160 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-hostroot\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.009160 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-daemon-config\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.009427 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-host\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.009427 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f83b2591-36a0-4341-b8d1-9a3b69f70275-env-overrides\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.009427 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7566bfc-2d5a-4514-b859-3879944bafa7-cni-binary-copy\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.009427 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-hostroot\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.009427 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83456891-bf2f-4d7d-82a6-cb2820c78575-serviceca\") pod \"node-ca-2859f\" (UID: \"83456891-bf2f-4d7d-82a6-cb2820c78575\") " pod="openshift-image-registry/node-ca-2859f"
Sep 09 20:16:21.009427 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v88wj\" (UniqueName: \"kubernetes.io/projected/e5088b6b-85b2-4c2f-b7b6-0f1039ba191a-kube-api-access-v88wj\") pod \"node-resolver-2xswt\" (UID: \"e5088b6b-85b2-4c2f-b7b6-0f1039ba191a\") " pod="openshift-dns/node-resolver-2xswt"
Sep 09 20:16:21.009427 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-cni-netd\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.009427 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:21.009427 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-kubernetes\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-systemd\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n726x\" (UniqueName: \"kubernetes.io/projected/f83b2591-36a0-4341-b8d1-9a3b69f70275-kube-api-access-n726x\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a63a3cd1-b73e-4524-8f2c-a94658df0401-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-host\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-run-multus-certs\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-kubernetes\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-modprobe-d\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-run-netns\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-run-multus-certs\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-device-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-systemd\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-sys\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009674    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-slash\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-node-log\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009703    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-modprobe-d\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzsfn\" (UniqueName: \"kubernetes.io/projected/a63a3cd1-b73e-4524-8f2c-a94658df0401-kube-api-access-qzsfn\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.009924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-etc-selinux\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009771    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-sys\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-sysctl-d\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009856    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adac9137-8ee2-40d6-9bea-9e9110d289d3-host-slash\") pod \"iptables-alerter-nzgb6\" (UID: \"adac9137-8ee2-40d6-9bea-9e9110d289d3\") " pod="openshift-network-operator/iptables-alerter-nzgb6"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-cni-dir\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-tmp\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-log-socket\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a63a3cd1-b73e-4524-8f2c-a94658df0401-cnibin\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3774603a-6336-4e25-82a3-c5a0b89fd082-kubelet-config\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3774603a-6336-4e25-82a3-c5a0b89fd082-dbus\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrdv\" (UniqueName: \"kubernetes.io/projected/704bf127-1228-454a-87bb-25b3eef70346-kube-api-access-mmrdv\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-run-netns\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlhk5\" (UniqueName: \"kubernetes.io/projected/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-kube-api-access-tlhk5\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-cni-dir\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-run-ovn\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.010902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-system-cni-dir\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-socket-dir-parent\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5088b6b-85b2-4c2f-b7b6-0f1039ba191a-tmp-dir\") pod \"node-resolver-2xswt\" (UID: \"e5088b6b-85b2-4c2f-b7b6-0f1039ba191a\") " pod="openshift-dns/node-resolver-2xswt"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f83b2591-36a0-4341-b8d1-9a3b69f70275-ovn-node-metrics-cert\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a63a3cd1-b73e-4524-8f2c-a94658df0401-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpv9w\" (UniqueName: \"kubernetes.io/projected/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-kube-api-access-hpv9w\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89368c79-9c74-4509-8588-0600a7e46763-agent-certs\") pod \"konnectivity-agent-sdghr\" (UID: \"89368c79-9c74-4509-8588-0600a7e46763\") " pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-daemon-config\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-lib-modules\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-run-netns\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010514    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-var-lib-cni-bin\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-sysctl-conf\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83456891-bf2f-4d7d-82a6-cb2820c78575-host\") pod \"node-ca-2859f\" (UID: \"83456891-bf2f-4d7d-82a6-cb2820c78575\") " pod="openshift-image-registry/node-ca-2859f"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-system-cni-dir\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-kubelet\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.011737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010654    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-socket-dir-parent\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adac9137-8ee2-40d6-9bea-9e9110d289d3-iptables-alerter-script\") pod \"iptables-alerter-nzgb6\" (UID: \"adac9137-8ee2-40d6-9bea-9e9110d289d3\") " pod="openshift-network-operator/iptables-alerter-nzgb6"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rzd\" (UniqueName: \"kubernetes.io/projected/adac9137-8ee2-40d6-9bea-9e9110d289d3-kube-api-access-g4rzd\") pod \"iptables-alerter-nzgb6\" (UID: \"adac9137-8ee2-40d6-9bea-9e9110d289d3\") " pod="openshift-network-operator/iptables-alerter-nzgb6"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-var-lib-cni-bin\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-sys-fs\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-cnibin\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.009934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-sysctl-d\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-lib-modules\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-etc-kubernetes\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-run\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-cnibin\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-var-lib-kubelet\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx2nl\" (UniqueName: \"kubernetes.io/projected/83456891-bf2f-4d7d-82a6-cb2820c78575-kube-api-access-wx2nl\") pod \"node-ca-2859f\" (UID: \"83456891-bf2f-4d7d-82a6-cb2820c78575\") " pod="openshift-image-registry/node-ca-2859f"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010948    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-etc-kubernetes\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.010988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-systemd-units\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-sysctl-conf\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011003    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-run\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.012710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-var-lib-kubelet\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-cni-bin\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a63a3cd1-b73e-4524-8f2c-a94658df0401-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrdcr\" (UniqueName: \"kubernetes.io/projected/b7566bfc-2d5a-4514-b859-3879944bafa7-kube-api-access-hrdcr\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-sysconfig\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5088b6b-85b2-4c2f-b7b6-0f1039ba191a-hosts-file\") pod \"node-resolver-2xswt\" (UID: \"e5088b6b-85b2-4c2f-b7b6-0f1039ba191a\") " pod="openshift-dns/node-resolver-2xswt"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-run-systemd\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-sysconfig\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89368c79-9c74-4509-8588-0600a7e46763-konnectivity-ca\") pod \"konnectivity-agent-sdghr\" (UID: \"89368c79-9c74-4509-8588-0600a7e46763\") " pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-var-lib-kubelet\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-conf-dir\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-var-lib-openvswitch\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-multus-conf-dir\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f83b2591-36a0-4341-b8d1-9a3b69f70275-ovnkube-script-lib\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a63a3cd1-b73e-4524-8f2c-a94658df0401-system-cni-dir\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a63a3cd1-b73e-4524-8f2c-a94658df0401-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.013689 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-var-lib-kubelet\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-socket-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7566bfc-2d5a-4514-b859-3879944bafa7-cni-binary-copy\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-run-k8s-cni-cncf-io\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-var-lib-cni-multus\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-run-k8s-cni-cncf-io\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-etc-openvswitch\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7566bfc-2d5a-4514-b859-3879944bafa7-host-var-lib-cni-multus\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-tuned\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4klm\" (UniqueName: \"kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm\") pod \"network-check-target-lf5rw\" (UID: \"9e614f95-303b-473d-b59b-544cc9a5ce53\") " pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-run-openvswitch\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f83b2591-36a0-4341-b8d1-9a3b69f70275-ovnkube-config\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.014611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.011845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-registration-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.015248 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.014701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-tmp\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.015248 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.014719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-etc-tuned\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.018186 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.017801    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:21.018186 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.017829    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:21.018186 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.017845    2581 projected.go:196] Error preparing data for projected volume kube-api-access-s4klm for pod openshift-network-diagnostics/network-check-target-lf5rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:21.018186 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.017927    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm podName:9e614f95-303b-473d-b59b-544cc9a5ce53 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:21.517903988 +0000 UTC m=+3.011114964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s4klm" (UniqueName: "kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm") pod "network-check-target-lf5rw" (UID: "9e614f95-303b-473d-b59b-544cc9a5ce53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:21.020505 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.020478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrdcr\" (UniqueName: \"kubernetes.io/projected/b7566bfc-2d5a-4514-b859-3879944bafa7-kube-api-access-hrdcr\") pod \"multus-dm9v2\" (UID: \"b7566bfc-2d5a-4514-b859-3879944bafa7\") " pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.020621 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.020524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlhk5\" (UniqueName: \"kubernetes.io/projected/1e3b3859-9f95-4e86-8e5c-56e1d38473e2-kube-api-access-tlhk5\") pod \"tuned-qdt69\" (UID: \"1e3b3859-9f95-4e86-8e5c-56e1d38473e2\") " pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.112153 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.112153 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adac9137-8ee2-40d6-9bea-9e9110d289d3-host-slash\") pod \"iptables-alerter-nzgb6\" (UID: \"adac9137-8ee2-40d6-9bea-9e9110d289d3\") " pod="openshift-network-operator/iptables-alerter-nzgb6"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-log-socket\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a63a3cd1-b73e-4524-8f2c-a94658df0401-cnibin\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3774603a-6336-4e25-82a3-c5a0b89fd082-kubelet-config\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3774603a-6336-4e25-82a3-c5a0b89fd082-dbus\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adac9137-8ee2-40d6-9bea-9e9110d289d3-host-slash\") pod \"iptables-alerter-nzgb6\" (UID: \"adac9137-8ee2-40d6-9bea-9e9110d289d3\") " pod="openshift-network-operator/iptables-alerter-nzgb6"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a63a3cd1-b73e-4524-8f2c-a94658df0401-cnibin\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3774603a-6336-4e25-82a3-c5a0b89fd082-dbus\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112369    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3774603a-6336-4e25-82a3-c5a0b89fd082-kubelet-config\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrdv\" (UniqueName: \"kubernetes.io/projected/704bf127-1228-454a-87bb-25b3eef70346-kube-api-access-mmrdv\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-log-socket\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-run-ovn\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-run-ovn\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-run-ovn-kubernetes\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.112456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5088b6b-85b2-4c2f-b7b6-0f1039ba191a-tmp-dir\") pod \"node-resolver-2xswt\" (UID: \"e5088b6b-85b2-4c2f-b7b6-0f1039ba191a\") " pod="openshift-dns/node-resolver-2xswt"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f83b2591-36a0-4341-b8d1-9a3b69f70275-ovn-node-metrics-cert\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a63a3cd1-b73e-4524-8f2c-a94658df0401-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpv9w\" (UniqueName: \"kubernetes.io/projected/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-kube-api-access-hpv9w\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89368c79-9c74-4509-8588-0600a7e46763-agent-certs\") pod \"konnectivity-agent-sdghr\" (UID: \"89368c79-9c74-4509-8588-0600a7e46763\") " pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83456891-bf2f-4d7d-82a6-cb2820c78575-host\") pod \"node-ca-2859f\" (UID: \"83456891-bf2f-4d7d-82a6-cb2820c78575\") " pod="openshift-image-registry/node-ca-2859f"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-kubelet\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adac9137-8ee2-40d6-9bea-9e9110d289d3-iptables-alerter-script\") pod \"iptables-alerter-nzgb6\" (UID: \"adac9137-8ee2-40d6-9bea-9e9110d289d3\") " pod="openshift-network-operator/iptables-alerter-nzgb6"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rzd\" (UniqueName: \"kubernetes.io/projected/adac9137-8ee2-40d6-9bea-9e9110d289d3-kube-api-access-g4rzd\") pod \"iptables-alerter-nzgb6\" (UID: \"adac9137-8ee2-40d6-9bea-9e9110d289d3\") " pod="openshift-network-operator/iptables-alerter-nzgb6"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-sys-fs\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2nl\" (UniqueName: \"kubernetes.io/projected/83456891-bf2f-4d7d-82a6-cb2820c78575-kube-api-access-wx2nl\") pod \"node-ca-2859f\" (UID: \"83456891-bf2f-4d7d-82a6-cb2820c78575\") " pod="openshift-image-registry/node-ca-2859f"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-systemd-units\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-cni-bin\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.113154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a63a3cd1-b73e-4524-8f2c-a94658df0401-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5088b6b-85b2-4c2f-b7b6-0f1039ba191a-hosts-file\") pod \"node-resolver-2xswt\" (UID: \"e5088b6b-85b2-4c2f-b7b6-0f1039ba191a\") " pod="openshift-dns/node-resolver-2xswt"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-run-systemd\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89368c79-9c74-4509-8588-0600a7e46763-konnectivity-ca\") pod \"konnectivity-agent-sdghr\" (UID: \"89368c79-9c74-4509-8588-0600a7e46763\") " pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-var-lib-openvswitch\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f83b2591-36a0-4341-b8d1-9a3b69f70275-ovnkube-script-lib\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5088b6b-85b2-4c2f-b7b6-0f1039ba191a-tmp-dir\") pod \"node-resolver-2xswt\" (UID: \"e5088b6b-85b2-4c2f-b7b6-0f1039ba191a\") " pod="openshift-dns/node-resolver-2xswt"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.112974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a63a3cd1-b73e-4524-8f2c-a94658df0401-system-cni-dir\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113005    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a63a3cd1-b73e-4524-8f2c-a94658df0401-system-cni-dir\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a63a3cd1-b73e-4524-8f2c-a94658df0401-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-socket-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-etc-openvswitch\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-run-openvswitch\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a63a3cd1-b73e-4524-8f2c-a94658df0401-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f83b2591-36a0-4341-b8d1-9a3b69f70275-ovnkube-config\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-registration-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.114039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f83b2591-36a0-4341-b8d1-9a3b69f70275-env-overrides\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83456891-bf2f-4d7d-82a6-cb2820c78575-serviceca\") pod \"node-ca-2859f\" (UID: \"83456891-bf2f-4d7d-82a6-cb2820c78575\") " pod="openshift-image-registry/node-ca-2859f"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v88wj\" (UniqueName: \"kubernetes.io/projected/e5088b6b-85b2-4c2f-b7b6-0f1039ba191a-kube-api-access-v88wj\") pod \"node-resolver-2xswt\" (UID: \"e5088b6b-85b2-4c2f-b7b6-0f1039ba191a\") " pod="openshift-dns/node-resolver-2xswt"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-cni-netd\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n726x\" (UniqueName: \"kubernetes.io/projected/f83b2591-36a0-4341-b8d1-9a3b69f70275-kube-api-access-n726x\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a63a3cd1-b73e-4524-8f2c-a94658df0401-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-run-netns\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-registration-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-device-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113551    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-cni-bin\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83456891-bf2f-4d7d-82a6-cb2820c78575-host\") pod \"node-ca-2859f\" (UID: \"83456891-bf2f-4d7d-82a6-cb2820c78575\") " pod="openshift-image-registry/node-ca-2859f"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-kubelet\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a63a3cd1-b73e-4524-8f2c-a94658df0401-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-socket-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-etc-openvswitch\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-run-openvswitch\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.114899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f83b2591-36a0-4341-b8d1-9a3b69f70275-env-overrides\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adac9137-8ee2-40d6-9bea-9e9110d289d3-iptables-alerter-script\") pod \"iptables-alerter-nzgb6\" (UID: \"adac9137-8ee2-40d6-9bea-9e9110d289d3\") " pod="openshift-network-operator/iptables-alerter-nzgb6"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114093    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-sys-fs\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-cni-netd\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f83b2591-36a0-4341-b8d1-9a3b69f70275-ovnkube-config\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.113463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-device-dir\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-slash\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-node-log\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzsfn\" (UniqueName: \"kubernetes.io/projected/a63a3cd1-b73e-4524-8f2c-a94658df0401-kube-api-access-qzsfn\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-etc-selinux\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.114320    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/704bf127-1228-454a-87bb-25b3eef70346-etc-selinux\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-slash\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-node-log\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.114438    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs podName:60ae6a25-16b2-48d6-98dd-dcdc41d99f7e nodeName:}" failed. No retries permitted until 2026-09-09 20:16:21.614397641 +0000 UTC m=+3.107608625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs") pod "network-metrics-daemon-rbt27" (UID: "60ae6a25-16b2-48d6-98dd-dcdc41d99f7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-host-run-netns\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.114321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-systemd-units\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.115767 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.115034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a63a3cd1-b73e-4524-8f2c-a94658df0401-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.116657 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.115061    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:21.116657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.115064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/83456891-bf2f-4d7d-82a6-cb2820c78575-serviceca\") pod \"node-ca-2859f\" (UID: \"83456891-bf2f-4d7d-82a6-cb2820c78575\") " pod="openshift-image-registry/node-ca-2859f"
Sep 09 20:16:21.116657 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.115121    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret podName:3774603a-6336-4e25-82a3-c5a0b89fd082 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:21.61510466 +0000 UTC m=+3.108315630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret") pod "global-pull-secret-syncer-2f9sh" (UID: "3774603a-6336-4e25-82a3-c5a0b89fd082") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:21.116657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.115120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e5088b6b-85b2-4c2f-b7b6-0f1039ba191a-hosts-file\") pod \"node-resolver-2xswt\" (UID: \"e5088b6b-85b2-4c2f-b7b6-0f1039ba191a\") " pod="openshift-dns/node-resolver-2xswt"
Sep 09 20:16:21.116657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.115162    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-var-lib-openvswitch\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.116657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.115175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f83b2591-36a0-4341-b8d1-9a3b69f70275-run-systemd\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.116657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.115478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a63a3cd1-b73e-4524-8f2c-a94658df0401-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.116657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.115740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f83b2591-36a0-4341-b8d1-9a3b69f70275-ovnkube-script-lib\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.116657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.115777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89368c79-9c74-4509-8588-0600a7e46763-konnectivity-ca\") pod \"konnectivity-agent-sdghr\" (UID: \"89368c79-9c74-4509-8588-0600a7e46763\") " pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:21.117669 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.117643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f83b2591-36a0-4341-b8d1-9a3b69f70275-ovn-node-metrics-cert\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.119964 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.119915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89368c79-9c74-4509-8588-0600a7e46763-agent-certs\") pod \"konnectivity-agent-sdghr\" (UID: \"89368c79-9c74-4509-8588-0600a7e46763\") " pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:21.136039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.132975    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpv9w\" (UniqueName: \"kubernetes.io/projected/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-kube-api-access-hpv9w\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:21.136039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.133477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrdv\" (UniqueName: \"kubernetes.io/projected/704bf127-1228-454a-87bb-25b3eef70346-kube-api-access-mmrdv\") pod \"aws-ebs-csi-driver-node-tpv97\" (UID: \"704bf127-1228-454a-87bb-25b3eef70346\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.140727 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.140692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzsfn\" (UniqueName: \"kubernetes.io/projected/a63a3cd1-b73e-4524-8f2c-a94658df0401-kube-api-access-qzsfn\") pod \"multus-additional-cni-plugins-n6hd2\" (UID: \"a63a3cd1-b73e-4524-8f2c-a94658df0401\") " pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.140913 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.140889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n726x\" (UniqueName: \"kubernetes.io/projected/f83b2591-36a0-4341-b8d1-9a3b69f70275-kube-api-access-n726x\") pod \"ovnkube-node-kpdtc\" (UID: \"f83b2591-36a0-4341-b8d1-9a3b69f70275\") " pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.140985 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.140873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2nl\" (UniqueName: \"kubernetes.io/projected/83456891-bf2f-4d7d-82a6-cb2820c78575-kube-api-access-wx2nl\") pod \"node-ca-2859f\" (UID: \"83456891-bf2f-4d7d-82a6-cb2820c78575\") " pod="openshift-image-registry/node-ca-2859f"
Sep 09 20:16:21.141971 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.141942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rzd\" (UniqueName: \"kubernetes.io/projected/adac9137-8ee2-40d6-9bea-9e9110d289d3-kube-api-access-g4rzd\") pod \"iptables-alerter-nzgb6\" (UID: \"adac9137-8ee2-40d6-9bea-9e9110d289d3\") " pod="openshift-network-operator/iptables-alerter-nzgb6"
Sep 09 20:16:21.142458 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.142424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v88wj\" (UniqueName: \"kubernetes.io/projected/e5088b6b-85b2-4c2f-b7b6-0f1039ba191a-kube-api-access-v88wj\") pod \"node-resolver-2xswt\" (UID: \"e5088b6b-85b2-4c2f-b7b6-0f1039ba191a\") " pod="openshift-dns/node-resolver-2xswt"
Sep 09 20:16:21.197105 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.197069    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:16:21.208957 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.208921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dm9v2"
Sep 09 20:16:21.217556 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.217525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qdt69"
Sep 09 20:16:21.225241 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:21.225203    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7566bfc_2d5a_4514_b859_3879944bafa7.slice/crio-1e75f5cee2aefaadca2b591a852ca60d9e890859e1304d30756f2f5cd4626522 WatchSource:0}: Error finding container 1e75f5cee2aefaadca2b591a852ca60d9e890859e1304d30756f2f5cd4626522: Status 404 returned error can't find the container with id 1e75f5cee2aefaadca2b591a852ca60d9e890859e1304d30756f2f5cd4626522
Sep 09 20:16:21.229142 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.229116    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:21.229973 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:21.229754    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3b3859_9f95_4e86_8e5c_56e1d38473e2.slice/crio-41577b505d0869e86abfbd0c044d21dad084a59c99ad698fe35a83b3bbaf33aa WatchSource:0}: Error finding container 41577b505d0869e86abfbd0c044d21dad084a59c99ad698fe35a83b3bbaf33aa: Status 404 returned error can't find the container with id 41577b505d0869e86abfbd0c044d21dad084a59c99ad698fe35a83b3bbaf33aa
Sep 09 20:16:21.237104 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.236908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2859f"
Sep 09 20:16:21.243254 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.243228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2xswt"
Sep 09 20:16:21.251265 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.251164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:21.253906 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:21.253864    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83456891_bf2f_4d7d_82a6_cb2820c78575.slice/crio-54db1a733b961d1a7bfe3749010eea61b3cfb74eabb3109c96e7a4f83d68e4aa WatchSource:0}: Error finding container 54db1a733b961d1a7bfe3749010eea61b3cfb74eabb3109c96e7a4f83d68e4aa: Status 404 returned error can't find the container with id 54db1a733b961d1a7bfe3749010eea61b3cfb74eabb3109c96e7a4f83d68e4aa
Sep 09 20:16:21.259859 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.259561    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6hd2"
Sep 09 20:16:21.263984 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:21.263957    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5088b6b_85b2_4c2f_b7b6_0f1039ba191a.slice/crio-f5231eab96cc197543f0a93d79c0aa5081d4e501948586c555c63960386f5247 WatchSource:0}: Error finding container f5231eab96cc197543f0a93d79c0aa5081d4e501948586c555c63960386f5247: Status 404 returned error can't find the container with id f5231eab96cc197543f0a93d79c0aa5081d4e501948586c555c63960386f5247
Sep 09 20:16:21.267943 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.267918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nzgb6"
Sep 09 20:16:21.268185 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:21.268154    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83b2591_36a0_4341_b8d1_9a3b69f70275.slice/crio-0a83858d6163c0adfd145d161fc6ed4c977bc7f9546f777d7c96d300a0803c52 WatchSource:0}: Error finding container 0a83858d6163c0adfd145d161fc6ed4c977bc7f9546f777d7c96d300a0803c52: Status 404 returned error can't find the container with id 0a83858d6163c0adfd145d161fc6ed4c977bc7f9546f777d7c96d300a0803c52
Sep 09 20:16:21.273710 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:21.273457    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63a3cd1_b73e_4524_8f2c_a94658df0401.slice/crio-845236fd9ed8a905e4e883a90780f246a0a235162eb9c0b9d85769513165186f WatchSource:0}: Error finding container 845236fd9ed8a905e4e883a90780f246a0a235162eb9c0b9d85769513165186f: Status 404 returned error can't find the container with id 845236fd9ed8a905e4e883a90780f246a0a235162eb9c0b9d85769513165186f
Sep 09 20:16:21.273710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.273540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97"
Sep 09 20:16:21.283026 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:21.282988    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadac9137_8ee2_40d6_9bea_9e9110d289d3.slice/crio-e7edda6a030729e657930b9f5c8e36714fa7e65a3c37d5899ce8fa1be74dd4a0 WatchSource:0}: Error finding container e7edda6a030729e657930b9f5c8e36714fa7e65a3c37d5899ce8fa1be74dd4a0: Status 404 returned error can't find the container with id e7edda6a030729e657930b9f5c8e36714fa7e65a3c37d5899ce8fa1be74dd4a0
Sep 09 20:16:21.288270 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:21.288228    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704bf127_1228_454a_87bb_25b3eef70346.slice/crio-5f09028e66afc416566c9c820625fd08b7fd51b8be0611444c01a80262c56434 WatchSource:0}: Error finding container 5f09028e66afc416566c9c820625fd08b7fd51b8be0611444c01a80262c56434: Status 404 returned error can't find the container with id 5f09028e66afc416566c9c820625fd08b7fd51b8be0611444c01a80262c56434
Sep 09 20:16:21.518785 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.518687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4klm\" (UniqueName: \"kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm\") pod \"network-check-target-lf5rw\" (UID: \"9e614f95-303b-473d-b59b-544cc9a5ce53\") " pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:21.518939 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.518892    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:21.518939 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.518923    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:21.518939 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.518937    2581 projected.go:196] Error preparing data for projected volume kube-api-access-s4klm for pod openshift-network-diagnostics/network-check-target-lf5rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:21.519044 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.519008    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm podName:9e614f95-303b-473d-b59b-544cc9a5ce53 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:22.518987291 +0000 UTC m=+4.012198256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4klm" (UniqueName: "kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm") pod "network-check-target-lf5rw" (UID: "9e614f95-303b-473d-b59b-544cc9a5ce53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:21.619918 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.619867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:21.620094 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.619950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:21.620094 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.620010    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:21.620094 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.620061    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:21.620248 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.620098    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret podName:3774603a-6336-4e25-82a3-c5a0b89fd082 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:22.620080724 +0000 UTC m=+4.113291711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret") pod "global-pull-secret-syncer-2f9sh" (UID: "3774603a-6336-4e25-82a3-c5a0b89fd082") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:21.620248 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:21.620115    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs podName:60ae6a25-16b2-48d6-98dd-dcdc41d99f7e nodeName:}" failed. No retries permitted until 2026-09-09 20:16:22.620107941 +0000 UTC m=+4.113318903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs") pod "network-metrics-daemon-rbt27" (UID: "60ae6a25-16b2-48d6-98dd-dcdc41d99f7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:21.961922 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.961821    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:11:20 +0000 UTC" deadline="2028-02-17 06:15:06.77257589 +0000 UTC"
Sep 09 20:16:21.961922 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:21.961862    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12609h58m44.810720335s"
Sep 09 20:16:22.006162 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.006120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97" event={"ID":"704bf127-1228-454a-87bb-25b3eef70346","Type":"ContainerStarted","Data":"5f09028e66afc416566c9c820625fd08b7fd51b8be0611444c01a80262c56434"}
Sep 09 20:16:22.007634 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.007599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hd2" event={"ID":"a63a3cd1-b73e-4524-8f2c-a94658df0401","Type":"ContainerStarted","Data":"845236fd9ed8a905e4e883a90780f246a0a235162eb9c0b9d85769513165186f"}
Sep 09 20:16:22.009008 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.008978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc" event={"ID":"f83b2591-36a0-4341-b8d1-9a3b69f70275","Type":"ContainerStarted","Data":"0a83858d6163c0adfd145d161fc6ed4c977bc7f9546f777d7c96d300a0803c52"}
Sep 09 20:16:22.010186 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.010160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2xswt" event={"ID":"e5088b6b-85b2-4c2f-b7b6-0f1039ba191a","Type":"ContainerStarted","Data":"f5231eab96cc197543f0a93d79c0aa5081d4e501948586c555c63960386f5247"}
Sep 09 20:16:22.011477 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.011448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qdt69" event={"ID":"1e3b3859-9f95-4e86-8e5c-56e1d38473e2","Type":"ContainerStarted","Data":"41577b505d0869e86abfbd0c044d21dad084a59c99ad698fe35a83b3bbaf33aa"}
Sep 09 20:16:22.013147 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.013119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nzgb6" event={"ID":"adac9137-8ee2-40d6-9bea-9e9110d289d3","Type":"ContainerStarted","Data":"e7edda6a030729e657930b9f5c8e36714fa7e65a3c37d5899ce8fa1be74dd4a0"}
Sep 09 20:16:22.014463 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.014441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2859f" event={"ID":"83456891-bf2f-4d7d-82a6-cb2820c78575","Type":"ContainerStarted","Data":"54db1a733b961d1a7bfe3749010eea61b3cfb74eabb3109c96e7a4f83d68e4aa"}
Sep 09 20:16:22.015793 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.015759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sdghr" event={"ID":"89368c79-9c74-4509-8588-0600a7e46763","Type":"ContainerStarted","Data":"0c0c358b98d1d3c2d3a34b16e144ce53137753ea8f348d483afac0fdd4f54df9"}
Sep 09 20:16:22.017249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.017228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dm9v2" event={"ID":"b7566bfc-2d5a-4514-b859-3879944bafa7","Type":"ContainerStarted","Data":"1e75f5cee2aefaadca2b591a852ca60d9e890859e1304d30756f2f5cd4626522"}
Sep 09 20:16:22.526711 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.526677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4klm\" (UniqueName: \"kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm\") pod \"network-check-target-lf5rw\" (UID: \"9e614f95-303b-473d-b59b-544cc9a5ce53\") " pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:22.526895 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:22.526871    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:22.526895 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:22.526900    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:22.527067 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:22.526915    2581 projected.go:196] Error preparing data for projected volume kube-api-access-s4klm for pod openshift-network-diagnostics/network-check-target-lf5rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:22.527067 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:22.526984    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm podName:9e614f95-303b-473d-b59b-544cc9a5ce53 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:24.526965857 +0000 UTC m=+6.020176835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4klm" (UniqueName: "kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm") pod "network-check-target-lf5rw" (UID: "9e614f95-303b-473d-b59b-544cc9a5ce53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:22.627864 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.627778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:22.627864 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.627839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:22.628086 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:22.627967    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:22.628086 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:22.628021    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs podName:60ae6a25-16b2-48d6-98dd-dcdc41d99f7e nodeName:}" failed. No retries permitted until 2026-09-09 20:16:24.628008026 +0000 UTC m=+6.121218987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs") pod "network-metrics-daemon-rbt27" (UID: "60ae6a25-16b2-48d6-98dd-dcdc41d99f7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:22.628086 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:22.628064    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:22.628238 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:22.628093    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret podName:3774603a-6336-4e25-82a3-c5a0b89fd082 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:24.628085982 +0000 UTC m=+6.121296943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret") pod "global-pull-secret-syncer-2f9sh" (UID: "3774603a-6336-4e25-82a3-c5a0b89fd082") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:22.962761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.962636    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:11:20 +0000 UTC" deadline="2028-04-28 21:02:52.717200336 +0000 UTC"
Sep 09 20:16:22.962761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.962667    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14328h46m29.754536494s"
Sep 09 20:16:22.997958 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.997473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:22.997958 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:22.997604    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:22.998168 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.998008    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:22.998168 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:22.998122    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:22.998286 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:22.998267    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:22.998750 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:22.998722    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:23.031808 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:23.031768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-122.ec2.internal" event={"ID":"2f1edbe788da0ad39b612586be095693","Type":"ContainerStarted","Data":"794209e45294ae074a00009f2bc293f1290f709be2c1890b222fb15de806f9e1"}
Sep 09 20:16:23.035574 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:23.035528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal" event={"ID":"423ca18fda886fc65626bc69f5433aaa","Type":"ContainerStarted","Data":"0b7ff5a278ecd11e80263d5ab04e05caeb4d72ec3f52df305613c1ec964ae836"}
Sep 09 20:16:23.079318 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:23.079224    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-122.ec2.internal" podStartSLOduration=3.079203728 podStartE2EDuration="3.079203728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:23.053314905 +0000 UTC m=+4.546525903" watchObservedRunningTime="2026-09-09 20:16:23.079203728 +0000 UTC m=+4.572414713"
Sep 09 20:16:24.546252 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:24.546211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4klm\" (UniqueName: \"kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm\") pod \"network-check-target-lf5rw\" (UID: \"9e614f95-303b-473d-b59b-544cc9a5ce53\") " pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:24.546827 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:24.546440    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:24.546827 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:24.546464    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:24.546827 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:24.546477    2581 projected.go:196] Error preparing data for projected volume kube-api-access-s4klm for pod openshift-network-diagnostics/network-check-target-lf5rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:24.546827 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:24.546538    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm podName:9e614f95-303b-473d-b59b-544cc9a5ce53 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:28.546518274 +0000 UTC m=+10.039729255 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4klm" (UniqueName: "kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm") pod "network-check-target-lf5rw" (UID: "9e614f95-303b-473d-b59b-544cc9a5ce53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:24.647739 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:24.647174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:24.647739 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:24.647258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:24.647739 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:24.647337    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:24.647739 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:24.647358    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:24.647739 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:24.647422    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs podName:60ae6a25-16b2-48d6-98dd-dcdc41d99f7e nodeName:}" failed. No retries permitted until 2026-09-09 20:16:28.647403444 +0000 UTC m=+10.140614421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs") pod "network-metrics-daemon-rbt27" (UID: "60ae6a25-16b2-48d6-98dd-dcdc41d99f7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:24.647739 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:24.647439    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret podName:3774603a-6336-4e25-82a3-c5a0b89fd082 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:28.647430904 +0000 UTC m=+10.140641866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret") pod "global-pull-secret-syncer-2f9sh" (UID: "3774603a-6336-4e25-82a3-c5a0b89fd082") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:24.999413 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:24.998492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:24.999413 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:24.998625    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:24.999413 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:24.999018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:24.999413 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:24.999123    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:24.999413 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:24.999177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:24.999413 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:24.999245    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:25.045101 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:25.044403    2581 generic.go:356] "Generic (PLEG): container finished" podID="423ca18fda886fc65626bc69f5433aaa" containerID="0b7ff5a278ecd11e80263d5ab04e05caeb4d72ec3f52df305613c1ec964ae836" exitCode=0
Sep 09 20:16:25.045101 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:25.044464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal" event={"ID":"423ca18fda886fc65626bc69f5433aaa","Type":"ContainerDied","Data":"0b7ff5a278ecd11e80263d5ab04e05caeb4d72ec3f52df305613c1ec964ae836"}
Sep 09 20:16:26.996675 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:26.996646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:26.996675 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:26.996674    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:26.997178 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:26.996645    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:26.997178 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:26.996794    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:26.997178 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:26.996908    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:26.997178 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:26.996980    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:28.580985 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:28.580943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4klm\" (UniqueName: \"kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm\") pod \"network-check-target-lf5rw\" (UID: \"9e614f95-303b-473d-b59b-544cc9a5ce53\") " pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:28.581443 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:28.581120    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:28.581443 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:28.581148    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:28.581443 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:28.581162    2581 projected.go:196] Error preparing data for projected volume kube-api-access-s4klm for pod openshift-network-diagnostics/network-check-target-lf5rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:28.581443 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:28.581218    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm podName:9e614f95-303b-473d-b59b-544cc9a5ce53 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:36.581204893 +0000 UTC m=+18.074415859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4klm" (UniqueName: "kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm") pod "network-check-target-lf5rw" (UID: "9e614f95-303b-473d-b59b-544cc9a5ce53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:28.682249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:28.682209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:28.682437 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:28.682286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:28.682437 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:28.682426    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:28.682564 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:28.682492    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs podName:60ae6a25-16b2-48d6-98dd-dcdc41d99f7e nodeName:}" failed. No retries permitted until 2026-09-09 20:16:36.682473162 +0000 UTC m=+18.175684129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs") pod "network-metrics-daemon-rbt27" (UID: "60ae6a25-16b2-48d6-98dd-dcdc41d99f7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:28.682884 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:28.682798    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:28.682884 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:28.682841    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret podName:3774603a-6336-4e25-82a3-c5a0b89fd082 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:36.682827215 +0000 UTC m=+18.176038191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret") pod "global-pull-secret-syncer-2f9sh" (UID: "3774603a-6336-4e25-82a3-c5a0b89fd082") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:28.997291 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:28.997213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:28.997291 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:28.997246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:28.997511 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:28.997356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:28.997511 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:28.997366    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:28.997511 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:28.997450    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:28.997724 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:28.997510    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:31.000124 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:30.999421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:31.000124 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:30.999532    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:31.000124 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:30.999872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:31.000124 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:30.999951    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:31.000124 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:30.999989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:31.000124 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:31.000052    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:32.060789 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.060702    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97" event={"ID":"704bf127-1228-454a-87bb-25b3eef70346","Type":"ContainerStarted","Data":"b9e6b72481c05d07a5961c22514f0dd10b7ed56ab5003d527b18e5620e3637ba"}
Sep 09 20:16:32.062732 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.062707    2581 generic.go:356] "Generic (PLEG): container finished" podID="a63a3cd1-b73e-4524-8f2c-a94658df0401" containerID="f6f64e771970f59189102e9ddf9b0d5672a2f1ebbdc9878ed7d80d0eea935b44" exitCode=0
Sep 09 20:16:32.062911 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.062779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hd2" event={"ID":"a63a3cd1-b73e-4524-8f2c-a94658df0401","Type":"ContainerDied","Data":"f6f64e771970f59189102e9ddf9b0d5672a2f1ebbdc9878ed7d80d0eea935b44"}
Sep 09 20:16:32.066333 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.066306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2xswt" event={"ID":"e5088b6b-85b2-4c2f-b7b6-0f1039ba191a","Type":"ContainerStarted","Data":"ce252f964f2cbe1e4c812252908e78fed4c6dba185d15c16db32d1beecc1fb5a"}
Sep 09 20:16:32.069938 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.069909    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qdt69" event={"ID":"1e3b3859-9f95-4e86-8e5c-56e1d38473e2","Type":"ContainerStarted","Data":"3eb82d74b48408f1bddaf1c03e1777e327ad6dc6169caa5176949bc3f7f2a681"}
Sep 09 20:16:32.072243 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.072223    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal" event={"ID":"423ca18fda886fc65626bc69f5433aaa","Type":"ContainerStarted","Data":"086d9e3555a1b18d9cddc4b1986c40db5cd96ddaa5ae7b24ae6c26dee8f1e375"}
Sep 09 20:16:32.074307 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.074282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2859f" event={"ID":"83456891-bf2f-4d7d-82a6-cb2820c78575","Type":"ContainerStarted","Data":"eb6ba148a8d8a4a6ce33a0dd23682476342c51bf741e77faf2bb3169ad85bafa"}
Sep 09 20:16:32.076261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.076238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sdghr" event={"ID":"89368c79-9c74-4509-8588-0600a7e46763","Type":"ContainerStarted","Data":"2b0a9de792d08cc5650a2489cde81b202bc9fb0262896f2ccbabd913ccab86af"}
Sep 09 20:16:32.106837 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.106779    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qdt69" podStartSLOduration=2.93667918 podStartE2EDuration="13.106759568s" totalImagesPullingTime="10.170080388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:32.106376189 +0000 UTC m=+13.599587174" watchObservedRunningTime="2026-09-09 20:16:32.106759568 +0000 UTC m=+13.599970562"
Sep 09 20:16:32.151520 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.151026    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sdghr" podStartSLOduration=2.971186701 podStartE2EDuration="13.151010825s" totalImagesPullingTime="10.179824124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:32.125051349 +0000 UTC m=+13.618262333" watchObservedRunningTime="2026-09-09 20:16:32.151010825 +0000 UTC m=+13.644221809"
Sep 09 20:16:32.151520 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.151116    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2859f" podStartSLOduration=2.980589171 podStartE2EDuration="13.151110296s" totalImagesPullingTime="10.170521125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:32.150763194 +0000 UTC m=+13.643974178" watchObservedRunningTime="2026-09-09 20:16:32.151110296 +0000 UTC m=+13.644321279"
Sep 09 20:16:32.199239 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.199149    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-122.ec2.internal" podStartSLOduration=12.199137673 podStartE2EDuration="12.199137673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:32.19762843 +0000 UTC m=+13.690839416" watchObservedRunningTime="2026-09-09 20:16:32.199137673 +0000 UTC m=+13.692348658"
Sep 09 20:16:32.245208 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.245158    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2xswt" podStartSLOduration=3.108885523 podStartE2EDuration="13.24513707s" totalImagesPullingTime="10.136251547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:32.244687027 +0000 UTC m=+13.737898014" watchObservedRunningTime="2026-09-09 20:16:32.24513707 +0000 UTC m=+13.738348054"
Sep 09 20:16:32.996854 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.996820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:32.997030 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.996821    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:32.997030 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:32.996965    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:32.997030 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:32.996936    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:32.997200 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:32.997062    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:32.997200 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:32.997139    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:33.079891 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:33.079856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nzgb6" event={"ID":"adac9137-8ee2-40d6-9bea-9e9110d289d3","Type":"ContainerStarted","Data":"c61edefb1132e62db0fe81f6abd6eaa9c11dc1e8780b45cc7cb95fd3af900138"}
Sep 09 20:16:33.096046 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:33.095986    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nzgb6" podStartSLOduration=4.707946832 podStartE2EDuration="14.095967746s" totalImagesPullingTime="9.388020914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:33.09542527 +0000 UTC m=+14.588636254" watchObservedRunningTime="2026-09-09 20:16:33.095967746 +0000 UTC m=+14.589178732"
Sep 09 20:16:35.000179 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:34.999952    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:35.000632 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:35.000276    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:35.000690 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:35.000639    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:35.000754 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:35.000724    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:35.000809 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:35.000769    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:35.000856 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:35.000845    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:35.492351 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:35.492249    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:35.616427 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:35.616388    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:35.616967 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:35.616950    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:36.087675 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:36.087647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sdghr"
Sep 09 20:16:36.652097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:36.652053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4klm\" (UniqueName: \"kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm\") pod \"network-check-target-lf5rw\" (UID: \"9e614f95-303b-473d-b59b-544cc9a5ce53\") " pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:36.652252 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:36.652197    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:36.652252 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:36.652215    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:36.652252 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:36.652224    2581 projected.go:196] Error preparing data for projected volume kube-api-access-s4klm for pod openshift-network-diagnostics/network-check-target-lf5rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:36.652391 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:36.652284    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm podName:9e614f95-303b-473d-b59b-544cc9a5ce53 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:52.652266099 +0000 UTC m=+34.145477065 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4klm" (UniqueName: "kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm") pod "network-check-target-lf5rw" (UID: "9e614f95-303b-473d-b59b-544cc9a5ce53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:36.753132 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:36.753078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:36.753132 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:36.753141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:36.753387 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:36.753245    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:36.753387 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:36.753250    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:36.753387 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:36.753303    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs podName:60ae6a25-16b2-48d6-98dd-dcdc41d99f7e nodeName:}" failed. No retries permitted until 2026-09-09 20:16:52.753286017 +0000 UTC m=+34.246496982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs") pod "network-metrics-daemon-rbt27" (UID: "60ae6a25-16b2-48d6-98dd-dcdc41d99f7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:36.753387 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:36.753323    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret podName:3774603a-6336-4e25-82a3-c5a0b89fd082 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:52.753314403 +0000 UTC m=+34.246525366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret") pod "global-pull-secret-syncer-2f9sh" (UID: "3774603a-6336-4e25-82a3-c5a0b89fd082") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:36.996699 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:36.996618    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:36.996699 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:36.996641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:36.996899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:36.996618    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:36.996899 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:36.996757    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:36.996899 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:36.996826    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:36.997047 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:36.996909    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:38.996790 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:38.996757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:38.997391 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:38.996841    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:38.997391 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:38.996866    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:38.997391 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:38.996906    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:38.997391 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:38.996988    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:38.997391 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:38.997078    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:40.996507 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:40.996458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:40.997073 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:40.996458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:40.997073 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:40.996580    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:40.997073 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:40.996471    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:40.997073 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:40.996715    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:40.997073 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:40.996805    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:42.101498 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:42.101302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dm9v2" event={"ID":"b7566bfc-2d5a-4514-b859-3879944bafa7","Type":"ContainerStarted","Data":"b019d63b088800f732caed7f9820f21ac62a9e12aed6743b161f41dae3daf9fe"}
Sep 09 20:16:42.103537 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:42.103504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc" event={"ID":"f83b2591-36a0-4341-b8d1-9a3b69f70275","Type":"ContainerStarted","Data":"b23994cb98271b2d4fcaf0dde7689129684a0f40ccdcd9a0ce2e0d15a81d225a"}
Sep 09 20:16:42.121071 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:42.121016    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dm9v2" podStartSLOduration=2.392765654 podStartE2EDuration="23.120988265s" totalImagesPullingTime="20.728222611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:42.120489136 +0000 UTC m=+23.613700119" watchObservedRunningTime="2026-09-09 20:16:42.120988265 +0000 UTC m=+23.614199249"
Sep 09 20:16:42.169578 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:42.169548    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:16:42.972618 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:42.972263    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:16:42.169568532Z","UUID":"d8edba11-f090-4acd-a863-fa713e53b74b","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:16:42.973617 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:42.973599    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:16:42.973731 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:42.973632    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:16:42.998947 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:42.998925    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:42.999046 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:42.998964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:42.999046 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:42.999034    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:42.999114 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:42.999084    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:42.999183 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:42.999166    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:42.999262 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:42.999244    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:43.107819 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:43.107785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97" event={"ID":"704bf127-1228-454a-87bb-25b3eef70346","Type":"ContainerStarted","Data":"8cc1dabde10fbde2dbad45ce51a6c55fc3f74f1236afe444cb9fc83b768bbd5f"}
Sep 09 20:16:43.109993 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:43.109963    2581 generic.go:356] "Generic (PLEG): container finished" podID="a63a3cd1-b73e-4524-8f2c-a94658df0401" containerID="0b4ed9571183081f8cd1e316f2582ecd41eb36f5c1287e1c03755d0fc5e987f8" exitCode=0
Sep 09 20:16:43.110134 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:43.110055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hd2" event={"ID":"a63a3cd1-b73e-4524-8f2c-a94658df0401","Type":"ContainerDied","Data":"0b4ed9571183081f8cd1e316f2582ecd41eb36f5c1287e1c03755d0fc5e987f8"}
Sep 09 20:16:43.114397 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:43.114373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc" event={"ID":"f83b2591-36a0-4341-b8d1-9a3b69f70275","Type":"ContainerStarted","Data":"ddc25b44dbafbda3f9f8a6895e5a98ebfb0e38237e218eee1c6a6c0723506635"}
Sep 09 20:16:43.114543 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:43.114448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc" event={"ID":"f83b2591-36a0-4341-b8d1-9a3b69f70275","Type":"ContainerStarted","Data":"cd946b2d7bccd82b5791c8a783eb763995e9dd91408a42b35dbfb400d123364f"}
Sep 09 20:16:43.114543 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:43.114489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc" event={"ID":"f83b2591-36a0-4341-b8d1-9a3b69f70275","Type":"ContainerStarted","Data":"8ce11c4539f3b682fea5d735af619fa5852a98fb9529305876c5f25d25eaf7f8"}
Sep 09 20:16:43.114543 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:43.114529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc" event={"ID":"f83b2591-36a0-4341-b8d1-9a3b69f70275","Type":"ContainerStarted","Data":"fd07d3b29bce345944ac67b6c9af2bf0c38077c33ce2119cbb7aa760b71f5441"}
Sep 09 20:16:43.114709 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:43.114558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc" event={"ID":"f83b2591-36a0-4341-b8d1-9a3b69f70275","Type":"ContainerStarted","Data":"3a42e1e93f9254d19c1334bb551206d851893e041638a76a1654c5c4e8e39c78"}
Sep 09 20:16:44.118434 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:44.118397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97" event={"ID":"704bf127-1228-454a-87bb-25b3eef70346","Type":"ContainerStarted","Data":"4d9d3553d52e55bb02b60b84a9002509a8d46de175e205fe1a3a5e080fc2e5bc"}
Sep 09 20:16:44.138624 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:44.138534    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tpv97" podStartSLOduration=3.145923811 podStartE2EDuration="25.138519136s" totalImagesPullingTime="21.992595325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:44.138266146 +0000 UTC m=+25.631477131" watchObservedRunningTime="2026-09-09 20:16:44.138519136 +0000 UTC m=+25.631730120"
Sep 09 20:16:44.999589 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:44.999513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:44.999746 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:44.999616    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:44.999746 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:44.999621    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:44.999746 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:44.999647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:44.999948 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:44.999747    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:44.999948 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:44.999845    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:45.122817 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:45.122786    2581 generic.go:356] "Generic (PLEG): container finished" podID="a63a3cd1-b73e-4524-8f2c-a94658df0401" containerID="5cce622d33a50ab30ad31bd93cc8a4a889aa0f4cb528e09a7456f1d7d9b94453" exitCode=0
Sep 09 20:16:45.123219 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:45.122870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hd2" event={"ID":"a63a3cd1-b73e-4524-8f2c-a94658df0401","Type":"ContainerDied","Data":"5cce622d33a50ab30ad31bd93cc8a4a889aa0f4cb528e09a7456f1d7d9b94453"}
Sep 09 20:16:45.126469 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:45.126443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc" event={"ID":"f83b2591-36a0-4341-b8d1-9a3b69f70275","Type":"ContainerStarted","Data":"9278d139b40cba92dce9afbe8128afd4b12e1055d78673e5e426916748749963"}
Sep 09 20:16:46.996704 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:46.996682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:46.997386 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:46.996690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:46.997386 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:46.996798    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:46.997386 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:46.996692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:46.997386 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:46.996932    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:46.997386 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:46.997040    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:47.133452 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:47.133421    2581 generic.go:356] "Generic (PLEG): container finished" podID="a63a3cd1-b73e-4524-8f2c-a94658df0401" containerID="4ddf463bd9c91ab87a75b54edb1b1d8304193e902af1b669ce4136a6bcffaea2" exitCode=0
Sep 09 20:16:47.133633 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:47.133513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hd2" event={"ID":"a63a3cd1-b73e-4524-8f2c-a94658df0401","Type":"ContainerDied","Data":"4ddf463bd9c91ab87a75b54edb1b1d8304193e902af1b669ce4136a6bcffaea2"}
Sep 09 20:16:47.138124 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:47.138100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc" event={"ID":"f83b2591-36a0-4341-b8d1-9a3b69f70275","Type":"ContainerStarted","Data":"068a2b069c90fee929aa2a7523f1a7c3888368677856c9d593321acb0e0ad035"}
Sep 09 20:16:47.138403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:47.138376    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:47.138616 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:47.138494    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:47.138616 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:47.138557    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:47.156940 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:47.156914    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:47.157216 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:47.157195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:16:47.192249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:47.192017    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc" podStartSLOduration=7.503757373 podStartE2EDuration="28.192001662s" totalImagesPullingTime="20.688244289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:47.190802282 +0000 UTC m=+28.684013266" watchObservedRunningTime="2026-09-09 20:16:47.192001662 +0000 UTC m=+28.685212645"
Sep 09 20:16:48.906908 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:48.906879    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2f9sh"]
Sep 09 20:16:48.907376 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:48.907005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:48.907376 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:48.907119    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:48.909452 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:48.909416    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lf5rw"]
Sep 09 20:16:48.909586 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:48.909519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:48.909695 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:48.909613    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:48.912476 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:48.912452    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rbt27"]
Sep 09 20:16:48.912630 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:48.912609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:48.912766 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:48.912741    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:49.996864 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:49.996831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:49.997313 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:49.996871    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:49.997313 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:49.996959    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:49.997313 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:49.997042    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:50.998608 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:50.998573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:50.999012 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:50.998721    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:51.996051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:51.995973    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:51.996051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:51.996012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:51.996218 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:51.996089    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:51.996218 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:51.996163    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:52.665453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:52.665418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4klm\" (UniqueName: \"kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm\") pod \"network-check-target-lf5rw\" (UID: \"9e614f95-303b-473d-b59b-544cc9a5ce53\") " pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:52.665800 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:52.665548    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:16:52.665800 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:52.665567    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:16:52.665800 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:52.665577    2581 projected.go:196] Error preparing data for projected volume kube-api-access-s4klm for pod openshift-network-diagnostics/network-check-target-lf5rw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:52.665800 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:52.665628    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm podName:9e614f95-303b-473d-b59b-544cc9a5ce53 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:24.6656161 +0000 UTC m=+66.158827061 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s4klm" (UniqueName: "kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm") pod "network-check-target-lf5rw" (UID: "9e614f95-303b-473d-b59b-544cc9a5ce53") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:16:52.766559 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:52.766530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:52.766698 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:52.766578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:52.766698 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:52.766667    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:52.766698 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:52.766668    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:52.766782 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:52.766715    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs podName:60ae6a25-16b2-48d6-98dd-dcdc41d99f7e nodeName:}" failed. No retries permitted until 2026-09-09 20:17:24.766703128 +0000 UTC m=+66.259914090 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs") pod "network-metrics-daemon-rbt27" (UID: "60ae6a25-16b2-48d6-98dd-dcdc41d99f7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:16:52.766782 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:52.766728    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret podName:3774603a-6336-4e25-82a3-c5a0b89fd082 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:24.7667223 +0000 UTC m=+66.259933261 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret") pod "global-pull-secret-syncer-2f9sh" (UID: "3774603a-6336-4e25-82a3-c5a0b89fd082") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:16:52.996828 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:52.996799    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:52.997000 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:52.996901    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:53.154239 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:53.154208    2581 generic.go:356] "Generic (PLEG): container finished" podID="a63a3cd1-b73e-4524-8f2c-a94658df0401" containerID="d7f0d3cfa1d0e2d771b0bccd86943dc246f0a93bc979637719bd917b93eb735f" exitCode=0
Sep 09 20:16:53.154386 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:53.154257    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hd2" event={"ID":"a63a3cd1-b73e-4524-8f2c-a94658df0401","Type":"ContainerDied","Data":"d7f0d3cfa1d0e2d771b0bccd86943dc246f0a93bc979637719bd917b93eb735f"}
Sep 09 20:16:53.996155 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:53.995979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:53.996615 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:53.995980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:53.996615 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:53.996220    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f9sh" podUID="3774603a-6336-4e25-82a3-c5a0b89fd082"
Sep 09 20:16:53.996615 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:53.996329    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lf5rw" podUID="9e614f95-303b-473d-b59b-544cc9a5ce53"
Sep 09 20:16:54.159166 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:54.159129    2581 generic.go:356] "Generic (PLEG): container finished" podID="a63a3cd1-b73e-4524-8f2c-a94658df0401" containerID="05da69b896bc304dbb458d3020bdc32d551cea2e8ad9ef773a6d1cd62104f419" exitCode=0
Sep 09 20:16:54.159331 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:54.159180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hd2" event={"ID":"a63a3cd1-b73e-4524-8f2c-a94658df0401","Type":"ContainerDied","Data":"05da69b896bc304dbb458d3020bdc32d551cea2e8ad9ef773a6d1cd62104f419"}
Sep 09 20:16:54.996452 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:54.996414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:54.996859 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:54.996561    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rbt27" podUID="60ae6a25-16b2-48d6-98dd-dcdc41d99f7e"
Sep 09 20:16:55.165209 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.165175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6hd2" event={"ID":"a63a3cd1-b73e-4524-8f2c-a94658df0401","Type":"ContainerStarted","Data":"71758962592b660df58a02b45a083d21826e56c9b488667fa2f03fcb08ef1c06"}
Sep 09 20:16:55.188799 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.188746    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n6hd2" podStartSLOduration=8.319002781 podStartE2EDuration="36.188730107s" totalImagesPullingTime="27.869727326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:55.187109142 +0000 UTC m=+36.680320126" watchObservedRunningTime="2026-09-09 20:16:55.188730107 +0000 UTC m=+36.681941088"
Sep 09 20:16:55.748806 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.748776    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-122.ec2.internal" event="NodeReady"
Sep 09 20:16:55.748970 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.748892    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:16:55.808662 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.808596    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84f9984f7d-8f8lz"]
Sep 09 20:16:55.845856 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.845815    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"]
Sep 09 20:16:55.846021 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.845961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:55.849170 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.849149    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:16:55.850610 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.850590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:16:55.850787 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.850767    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:16:55.850913 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.850893    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qrtsj\""
Sep 09 20:16:55.856601 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.856581    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-7c664"]
Sep 09 20:16:55.856703 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.856690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:16:55.862258 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.862237    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 20:16:55.863158 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.863138    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:16:55.863228 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.863175    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 20:16:55.865365 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.865308    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c7xds\""
Sep 09 20:16:55.865473 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.865316    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:16:55.875441 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.875413    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"]
Sep 09 20:16:55.875560 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.875533    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:55.878662 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.878643    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:16:55.878894 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.878876    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 20:16:55.879114 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.879097    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 20:16:55.879206 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.879187    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-snvkv\""
Sep 09 20:16:55.879320 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.879305    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 20:16:55.884022 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.883999    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 20:16:55.900208 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.900182    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"]
Sep 09 20:16:55.900319 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.900289    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:55.902808 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.902790    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 20:16:55.907129 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.907108    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hxwhc\""
Sep 09 20:16:55.907253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.907113    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:16:55.907253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.907115    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:16:55.907253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.907216    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 20:16:55.916311 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.916288    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-gx45b"]
Sep 09 20:16:55.916426 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.916401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"
Sep 09 20:16:55.921106 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.921085    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 20:16:55.921227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.921089    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 20:16:55.921227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.921192    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:16:55.921227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.921213    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 20:16:55.921500 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.921484    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-29x9h\""
Sep 09 20:16:55.937658 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.937632    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"]
Sep 09 20:16:55.937773 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.937758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:55.943303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.943207    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 20:16:55.943534 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.943207    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 20:16:55.943678 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.943661    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:16:55.943766 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.943682    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4gbt4\""
Sep 09 20:16:55.944614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.944599    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:16:55.947989 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.947974    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 20:16:55.953810 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.953791    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-c57sd"]
Sep 09 20:16:55.953920 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.953905    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"
Sep 09 20:16:55.956291 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.956270    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 20:16:55.956291 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.956287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jggks\""
Sep 09 20:16:55.956449 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.956290    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:16:55.956449 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.956363    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 20:16:55.956449 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.956387    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 20:16:55.963781 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.963765    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-gx45b"]
Sep 09 20:16:55.963855 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.963791    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f9984f7d-8f8lz"]
Sep 09 20:16:55.963855 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.963806    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"]
Sep 09 20:16:55.963855 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.963820    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"]
Sep 09 20:16:55.963855 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.963837    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-p5rkt"]
Sep 09 20:16:55.964046 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.963875    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-c57sd"
Sep 09 20:16:55.966715 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.966699    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 20:16:55.966836 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.966820    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:16:55.966897 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.966890    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ddrp4\""
Sep 09 20:16:55.977834 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.977816    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c4444dc76-c2dcn"]
Sep 09 20:16:55.977958 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.977943    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-p5rkt"
Sep 09 20:16:55.980533 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.980515    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:16:55.980604 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.980541    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:16:55.980604 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.980520    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-z8n9b\""
Sep 09 20:16:55.989932 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.989911    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"]
Sep 09 20:16:55.990048 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.989954    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-c57sd"]
Sep 09 20:16:55.990048 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.989976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-7c664"]
Sep 09 20:16:55.990048 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.989996    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"]
Sep 09 20:16:55.990048 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990016    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-p5rkt"]
Sep 09 20:16:55.990048 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990033    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c4444dc76-c2dcn"]
Sep 09 20:16:55.990276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:55.990276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-config\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:55.990276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:55.990276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-serving-cert\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:55.990276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990140    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-registry-certificates\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:55.990276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45tsw\" (UniqueName: \"kubernetes.io/projected/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-kube-api-access-45tsw\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:55.990276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-trusted-ca\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:55.990276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990053    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t552h"]
Sep 09 20:16:55.990276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306d365e-c1ae-4944-868f-844b11fd59f4-ca-trust-extracted\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:55.990276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-installation-pull-secrets\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:55.990276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58vv\" (UniqueName: \"kubernetes.io/projected/bad2e819-d7a0-4f3a-bbd0-57b076806090-kube-api-access-q58vv\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:16:55.990752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-trusted-ca\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:55.990752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/670f84b1-5f23-44cf-9b05-e4a3e603b7fa-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tbhnt\" (UID: \"670f84b1-5f23-44cf-9b05-e4a3e603b7fa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"
Sep 09 20:16:55.990752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990406    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2mp4\" (UniqueName: \"kubernetes.io/projected/cc4d0a36-bc84-4ffe-b5d8-804d35114373-kube-api-access-k2mp4\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:55.990752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbhm\" (UniqueName: \"kubernetes.io/projected/670f84b1-5f23-44cf-9b05-e4a3e603b7fa-kube-api-access-5dbhm\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tbhnt\" (UID: \"670f84b1-5f23-44cf-9b05-e4a3e603b7fa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"
Sep 09 20:16:55.990752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/670f84b1-5f23-44cf-9b05-e4a3e603b7fa-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tbhnt\" (UID: \"670f84b1-5f23-44cf-9b05-e4a3e603b7fa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"
Sep 09 20:16:55.990752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7vkj\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-kube-api-access-f7vkj\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:55.990752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc4d0a36-bc84-4ffe-b5d8-804d35114373-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:55.990752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-image-registry-private-configuration\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:55.990752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-bound-sa-token\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:55.990752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:55.990752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.990632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:16:55.993413 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.993395    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 20:16:55.993520 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.993505    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:16:55.993582 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.993520    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 20:16:55.993740 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.993726    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:16:55.993808 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.993726    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-blkcn\""
Sep 09 20:16:55.993856 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.993801    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 20:16:55.993856 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:55.993817    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 20:16:56.002138 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.002081    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z7wr5"]
Sep 09 20:16:56.002578 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.002185    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t552h"
Sep 09 20:16:56.002578 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.002277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:16:56.002578 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.002191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:16:56.004890 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.004873    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:16:56.005130 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.005115    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:16:56.005202 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.005138    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:16:56.005253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.005200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mt874\""
Sep 09 20:16:56.005253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.005204    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c88hn\""
Sep 09 20:16:56.005477 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.005460    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:16:56.013934 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.013915    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t552h"]
Sep 09 20:16:56.014015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.013948    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z7wr5"]
Sep 09 20:16:56.014449 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.014434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.016903 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.016882    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:16:56.017149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.017132    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8pv6p\""
Sep 09 20:16:56.017286 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.017267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:16:56.091664 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.091637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vkj\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-kube-api-access-f7vkj\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.091664 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.091667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc4d0a36-bc84-4ffe-b5d8-804d35114373-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:56.091931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.091692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7npn\" (UniqueName: \"kubernetes.io/projected/749e9730-db41-43f8-914b-a435c9d671ff-kube-api-access-x7npn\") pod \"network-check-source-685456655f-p5rkt\" (UID: \"749e9730-db41-43f8-914b-a435c9d671ff\") " pod="openshift-network-diagnostics/network-check-source-685456655f-p5rkt"
Sep 09 20:16:56.091931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.091711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:16:56.091931 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.091794    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:16:56.091931 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.091855    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls podName:bad2e819-d7a0-4f3a-bbd0-57b076806090 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:56.591839039 +0000 UTC m=+38.085050001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cwptv" (UID: "bad2e819-d7a0-4f3a-bbd0-57b076806090") : secret "samples-operator-tls" not found
Sep 09 20:16:56.091931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.091867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-image-registry-private-configuration\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.091931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.091889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83a81b0e-8d00-470d-b1b3-28b2832b2746-serving-cert\") pod \"service-ca-operator-859f94c4b-psxc9\" (UID: \"83a81b0e-8d00-470d-b1b3-28b2832b2746\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"
Sep 09 20:16:56.091931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.091911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbp4\" (UniqueName: \"kubernetes.io/projected/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-kube-api-access-dtbp4\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.091931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.091933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83a81b0e-8d00-470d-b1b3-28b2832b2746-config\") pod \"service-ca-operator-859f94c4b-psxc9\" (UID: \"83a81b0e-8d00-470d-b1b3-28b2832b2746\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"
Sep 09 20:16:56.092303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.091971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:56.092303 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.092027    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:16:56.092303 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.092053    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls podName:cc4d0a36-bc84-4ffe-b5d8-804d35114373 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:56.592044099 +0000 UTC m=+38.085255061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-mfr42" (UID: "cc4d0a36-bc84-4ffe-b5d8-804d35114373") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:16:56.092303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-registry-certificates\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.092303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9wbq\" (UniqueName: \"kubernetes.io/projected/83a81b0e-8d00-470d-b1b3-28b2832b2746-kube-api-access-w9wbq\") pod \"service-ca-operator-859f94c4b-psxc9\" (UID: \"83a81b0e-8d00-470d-b1b3-28b2832b2746\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"
Sep 09 20:16:56.092303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-stats-auth\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.092303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnpl\" (UniqueName: \"kubernetes.io/projected/2be67041-c131-4335-a192-7476a5fcdbf1-kube-api-access-zwnpl\") pod \"volume-data-source-validator-695c895bd6-c57sd\" (UID: \"2be67041-c131-4335-a192-7476a5fcdbf1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-c57sd"
Sep 09 20:16:56.092303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306d365e-c1ae-4944-868f-844b11fd59f4-ca-trust-extracted\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.092303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pgsv\" (UniqueName: \"kubernetes.io/projected/a32e9cee-8743-486e-a042-41ae301e7f11-kube-api-access-6pgsv\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.092303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58vv\" (UniqueName: \"kubernetes.io/projected/bad2e819-d7a0-4f3a-bbd0-57b076806090-kube-api-access-q58vv\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:16:56.092303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092235    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-tmp-dir\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dab7b49e-a152-44f0-8ebb-3643d8862b0c-cert\") pod \"ingress-canary-t552h\" (UID: \"dab7b49e-a152-44f0-8ebb-3643d8862b0c\") " pod="openshift-ingress-canary/ingress-canary-t552h"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2mp4\" (UniqueName: \"kubernetes.io/projected/cc4d0a36-bc84-4ffe-b5d8-804d35114373-kube-api-access-k2mp4\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/670f84b1-5f23-44cf-9b05-e4a3e603b7fa-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tbhnt\" (UID: \"670f84b1-5f23-44cf-9b05-e4a3e603b7fa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cc4d0a36-bc84-4ffe-b5d8-804d35114373-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a32e9cee-8743-486e-a042-41ae301e7f11-serving-cert\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-bound-sa-token\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306d365e-c1ae-4944-868f-844b11fd59f4-ca-trust-extracted\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-config-volume\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-registry-certificates\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32e9cee-8743-486e-a042-41ae301e7f11-tmp\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32e9cee-8743-486e-a042-41ae301e7f11-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.092853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-config\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:56.093642 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.092832    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:16:56.093642 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.092846    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9984f7d-8f8lz: secret "image-registry-tls" not found
Sep 09 20:16:56.093642 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.092895    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls podName:306d365e-c1ae-4944-868f-844b11fd59f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:56.59288101 +0000 UTC m=+38.086091998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls") pod "image-registry-84f9984f7d-8f8lz" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4") : secret "image-registry-tls" not found
Sep 09 20:16:56.093642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-serving-cert\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:56.093642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45tsw\" (UniqueName: \"kubernetes.io/projected/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-kube-api-access-45tsw\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:56.093642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.092973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-trusted-ca\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:56.093642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.093007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a32e9cee-8743-486e-a042-41ae301e7f11-snapshots\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.093642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.093031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-installation-pull-secrets\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.093642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.093544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-config\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:56.093642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.093600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-default-certificate\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.093642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.093638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-trusted-ca\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.094097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.093665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/670f84b1-5f23-44cf-9b05-e4a3e603b7fa-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tbhnt\" (UID: \"670f84b1-5f23-44cf-9b05-e4a3e603b7fa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"
Sep 09 20:16:56.094097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.093695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbhm\" (UniqueName: \"kubernetes.io/projected/670f84b1-5f23-44cf-9b05-e4a3e603b7fa-kube-api-access-5dbhm\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tbhnt\" (UID: \"670f84b1-5f23-44cf-9b05-e4a3e603b7fa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"
Sep 09 20:16:56.094097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.093724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljhhb\" (UniqueName: \"kubernetes.io/projected/23891ba7-4710-4468-b69d-8764b3cf8384-kube-api-access-ljhhb\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.094097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.093750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32e9cee-8743-486e-a042-41ae301e7f11-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.094097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.093776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfvtk\" (UniqueName: \"kubernetes.io/projected/dab7b49e-a152-44f0-8ebb-3643d8862b0c-kube-api-access-pfvtk\") pod \"ingress-canary-t552h\" (UID: \"dab7b49e-a152-44f0-8ebb-3643d8862b0c\") " pod="openshift-ingress-canary/ingress-canary-t552h"
Sep 09 20:16:56.094097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.093929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-trusted-ca\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:56.094422 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.094402    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/670f84b1-5f23-44cf-9b05-e4a3e603b7fa-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tbhnt\" (UID: \"670f84b1-5f23-44cf-9b05-e4a3e603b7fa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"
Sep 09 20:16:56.094503 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.094484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-trusted-ca\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.096418 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.096397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/670f84b1-5f23-44cf-9b05-e4a3e603b7fa-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tbhnt\" (UID: \"670f84b1-5f23-44cf-9b05-e4a3e603b7fa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"
Sep 09 20:16:56.096542 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.096511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-serving-cert\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:56.096599 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.096568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-installation-pull-secrets\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.096652 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.096597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-image-registry-private-configuration\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.108320 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.108298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbhm\" (UniqueName: \"kubernetes.io/projected/670f84b1-5f23-44cf-9b05-e4a3e603b7fa-kube-api-access-5dbhm\") pod \"kube-storage-version-migrator-operator-8d7dd4675-tbhnt\" (UID: \"670f84b1-5f23-44cf-9b05-e4a3e603b7fa\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"
Sep 09 20:16:56.110667 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.110645    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vkj\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-kube-api-access-f7vkj\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.110789 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.110720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45tsw\" (UniqueName: \"kubernetes.io/projected/e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68-kube-api-access-45tsw\") pod \"console-operator-85b454874b-7c664\" (UID: \"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68\") " pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:56.110789 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.110723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58vv\" (UniqueName: \"kubernetes.io/projected/bad2e819-d7a0-4f3a-bbd0-57b076806090-kube-api-access-q58vv\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:16:56.110937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.110845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2mp4\" (UniqueName: \"kubernetes.io/projected/cc4d0a36-bc84-4ffe-b5d8-804d35114373-kube-api-access-k2mp4\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:56.111409 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.111392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-bound-sa-token\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.184702 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.184661    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:16:56.194938 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.194909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a32e9cee-8743-486e-a042-41ae301e7f11-serving-cert\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.195021 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.194942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-config-volume\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.195021 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.194960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.195021 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.194976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32e9cee-8743-486e-a042-41ae301e7f11-tmp\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.195021 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32e9cee-8743-486e-a042-41ae301e7f11-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.195215 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a32e9cee-8743-486e-a042-41ae301e7f11-snapshots\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.195215 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-default-certificate\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.195215 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljhhb\" (UniqueName: \"kubernetes.io/projected/23891ba7-4710-4468-b69d-8764b3cf8384-kube-api-access-ljhhb\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.195215 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.195130    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:56.695107637 +0000 UTC m=+38.188318615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : configmap references non-existent config key: service-ca.crt
Sep 09 20:16:56.195215 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32e9cee-8743-486e-a042-41ae301e7f11-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.195215 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfvtk\" (UniqueName: \"kubernetes.io/projected/dab7b49e-a152-44f0-8ebb-3643d8862b0c-kube-api-access-pfvtk\") pod \"ingress-canary-t552h\" (UID: \"dab7b49e-a152-44f0-8ebb-3643d8862b0c\") " pod="openshift-ingress-canary/ingress-canary-t552h"
Sep 09 20:16:56.195553 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7npn\" (UniqueName: \"kubernetes.io/projected/749e9730-db41-43f8-914b-a435c9d671ff-kube-api-access-x7npn\") pod \"network-check-source-685456655f-p5rkt\" (UID: \"749e9730-db41-43f8-914b-a435c9d671ff\") " pod="openshift-network-diagnostics/network-check-source-685456655f-p5rkt"
Sep 09 20:16:56.195553 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83a81b0e-8d00-470d-b1b3-28b2832b2746-serving-cert\") pod \"service-ca-operator-859f94c4b-psxc9\" (UID: \"83a81b0e-8d00-470d-b1b3-28b2832b2746\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"
Sep 09 20:16:56.195553 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbp4\" (UniqueName: \"kubernetes.io/projected/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-kube-api-access-dtbp4\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.195553 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83a81b0e-8d00-470d-b1b3-28b2832b2746-config\") pod \"service-ca-operator-859f94c4b-psxc9\" (UID: \"83a81b0e-8d00-470d-b1b3-28b2832b2746\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"
Sep 09 20:16:56.195553 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195423    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a32e9cee-8743-486e-a042-41ae301e7f11-tmp\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.195553 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9wbq\" (UniqueName: \"kubernetes.io/projected/83a81b0e-8d00-470d-b1b3-28b2832b2746-kube-api-access-w9wbq\") pod \"service-ca-operator-859f94c4b-psxc9\" (UID: \"83a81b0e-8d00-470d-b1b3-28b2832b2746\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"
Sep 09 20:16:56.195553 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-stats-auth\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.195553 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnpl\" (UniqueName: \"kubernetes.io/projected/2be67041-c131-4335-a192-7476a5fcdbf1-kube-api-access-zwnpl\") pod \"volume-data-source-validator-695c895bd6-c57sd\" (UID: \"2be67041-c131-4335-a192-7476a5fcdbf1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-c57sd"
Sep 09 20:16:56.195553 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pgsv\" (UniqueName: \"kubernetes.io/projected/a32e9cee-8743-486e-a042-41ae301e7f11-kube-api-access-6pgsv\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.196051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-tmp-dir\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.196051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-config-volume\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.196051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.196051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dab7b49e-a152-44f0-8ebb-3643d8862b0c-cert\") pod \"ingress-canary-t552h\" (UID: \"dab7b49e-a152-44f0-8ebb-3643d8862b0c\") " pod="openshift-ingress-canary/ingress-canary-t552h"
Sep 09 20:16:56.196051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.196051 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.195671    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:16:56.196051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.195716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a32e9cee-8743-486e-a042-41ae301e7f11-snapshots\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.196051 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.195726    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:56.695711166 +0000 UTC m=+38.188922128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : secret "router-metrics-certs-default" not found
Sep 09 20:16:56.196051 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.195764    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:16:56.196051 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.195804    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls podName:48d0e306-fdd9-4e95-ab7c-5ebdbb06f444 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:56.695795063 +0000 UTC m=+38.189006040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls") pod "dns-default-z7wr5" (UID: "48d0e306-fdd9-4e95-ab7c-5ebdbb06f444") : secret "dns-default-metrics-tls" not found
Sep 09 20:16:56.196594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.196119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32e9cee-8743-486e-a042-41ae301e7f11-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.196594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.196332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a32e9cee-8743-486e-a042-41ae301e7f11-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.196872 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.196841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/83a81b0e-8d00-470d-b1b3-28b2832b2746-config\") pod \"service-ca-operator-859f94c4b-psxc9\" (UID: \"83a81b0e-8d00-470d-b1b3-28b2832b2746\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"
Sep 09 20:16:56.197206 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.197186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-tmp-dir\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.197930 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.197907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a32e9cee-8743-486e-a042-41ae301e7f11-serving-cert\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.198310 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.198276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-default-certificate\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.198539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.198521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/83a81b0e-8d00-470d-b1b3-28b2832b2746-serving-cert\") pod \"service-ca-operator-859f94c4b-psxc9\" (UID: \"83a81b0e-8d00-470d-b1b3-28b2832b2746\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"
Sep 09 20:16:56.198603 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.198580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dab7b49e-a152-44f0-8ebb-3643d8862b0c-cert\") pod \"ingress-canary-t552h\" (UID: \"dab7b49e-a152-44f0-8ebb-3643d8862b0c\") " pod="openshift-ingress-canary/ingress-canary-t552h"
Sep 09 20:16:56.198603 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.198584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-stats-auth\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.209699 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.209664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnpl\" (UniqueName: \"kubernetes.io/projected/2be67041-c131-4335-a192-7476a5fcdbf1-kube-api-access-zwnpl\") pod \"volume-data-source-validator-695c895bd6-c57sd\" (UID: \"2be67041-c131-4335-a192-7476a5fcdbf1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-c57sd"
Sep 09 20:16:56.209799 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.209719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9wbq\" (UniqueName: \"kubernetes.io/projected/83a81b0e-8d00-470d-b1b3-28b2832b2746-kube-api-access-w9wbq\") pod \"service-ca-operator-859f94c4b-psxc9\" (UID: \"83a81b0e-8d00-470d-b1b3-28b2832b2746\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"
Sep 09 20:16:56.210149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.210126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbp4\" (UniqueName: \"kubernetes.io/projected/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-kube-api-access-dtbp4\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.210510 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.210488    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pgsv\" (UniqueName: \"kubernetes.io/projected/a32e9cee-8743-486e-a042-41ae301e7f11-kube-api-access-6pgsv\") pod \"insights-operator-76bcd6ccf5-gx45b\" (UID: \"a32e9cee-8743-486e-a042-41ae301e7f11\") " pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.210665 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.210643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7npn\" (UniqueName: \"kubernetes.io/projected/749e9730-db41-43f8-914b-a435c9d671ff-kube-api-access-x7npn\") pod \"network-check-source-685456655f-p5rkt\" (UID: \"749e9730-db41-43f8-914b-a435c9d671ff\") " pod="openshift-network-diagnostics/network-check-source-685456655f-p5rkt"
Sep 09 20:16:56.211616 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.211591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljhhb\" (UniqueName: \"kubernetes.io/projected/23891ba7-4710-4468-b69d-8764b3cf8384-kube-api-access-ljhhb\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.211785 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.211770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfvtk\" (UniqueName: \"kubernetes.io/projected/dab7b49e-a152-44f0-8ebb-3643d8862b0c-kube-api-access-pfvtk\") pod \"ingress-canary-t552h\" (UID: \"dab7b49e-a152-44f0-8ebb-3643d8862b0c\") " pod="openshift-ingress-canary/ingress-canary-t552h"
Sep 09 20:16:56.224700 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.224414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"
Sep 09 20:16:56.245594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.245566    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b"
Sep 09 20:16:56.261394 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.261377    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"
Sep 09 20:16:56.271569 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.271502    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-c57sd"
Sep 09 20:16:56.286829 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.286795    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-p5rkt"
Sep 09 20:16:56.312533 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.312492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t552h"
Sep 09 20:16:56.513564 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.513308    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-7c664"]
Sep 09 20:16:56.516869 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:56.516845    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ef9dbb_ce20_4e5c_9eb4_5f94a7c2cf68.slice/crio-d4a0aca08d88270d554fb0eacb3e1a58b37ddc255c84f9dca77ece14782b4216 WatchSource:0}: Error finding container d4a0aca08d88270d554fb0eacb3e1a58b37ddc255c84f9dca77ece14782b4216: Status 404 returned error can't find the container with id d4a0aca08d88270d554fb0eacb3e1a58b37ddc255c84f9dca77ece14782b4216
Sep 09 20:16:56.561770 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.561742    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt"]
Sep 09 20:16:56.564673 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:56.564634    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670f84b1_5f23_44cf_9b05_e4a3e603b7fa.slice/crio-f178af0b9870361dbf1b674ce6c6a5a8f4ccac866d089702b143c27c86623658 WatchSource:0}: Error finding container f178af0b9870361dbf1b674ce6c6a5a8f4ccac866d089702b143c27c86623658: Status 404 returned error can't find the container with id f178af0b9870361dbf1b674ce6c6a5a8f4ccac866d089702b143c27c86623658
Sep 09 20:16:56.596901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.596844    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9"]
Sep 09 20:16:56.597683 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.597643    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-gx45b"]
Sep 09 20:16:56.599915 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.599883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:56.600025 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.599990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:16:56.600025 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:56.600007    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a81b0e_8d00_470d_b1b3_28b2832b2746.slice/crio-eaa70f4a9ae35bf4efb1fba8bcba429f63652774ebf7c6282b096b4f458c106c WatchSource:0}: Error finding container eaa70f4a9ae35bf4efb1fba8bcba429f63652774ebf7c6282b096b4f458c106c: Status 404 returned error can't find the container with id eaa70f4a9ae35bf4efb1fba8bcba429f63652774ebf7c6282b096b4f458c106c
Sep 09 20:16:56.600137 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.600030    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:16:56.600137 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.600043    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9984f7d-8f8lz: secret "image-registry-tls" not found
Sep 09 20:16:56.600137 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.600040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:56.600137 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.600086    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:16:56.600137 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.600093    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls podName:306d365e-c1ae-4944-868f-844b11fd59f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:57.600075789 +0000 UTC m=+39.093286753 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls") pod "image-registry-84f9984f7d-8f8lz" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4") : secret "image-registry-tls" not found
Sep 09 20:16:56.600137 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.600129    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls podName:bad2e819-d7a0-4f3a-bbd0-57b076806090 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:57.600116027 +0000 UTC m=+39.093326992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cwptv" (UID: "bad2e819-d7a0-4f3a-bbd0-57b076806090") : secret "samples-operator-tls" not found
Sep 09 20:16:56.600441 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.600147    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:16:56.600441 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.600200    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls podName:cc4d0a36-bc84-4ffe-b5d8-804d35114373 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:57.600185826 +0000 UTC m=+39.093396808 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-mfr42" (UID: "cc4d0a36-bc84-4ffe-b5d8-804d35114373") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:16:56.600798 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:56.600703    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32e9cee_8743_486e_a042_41ae301e7f11.slice/crio-3b4b436048fa83ec8ef58cefd04102eb4ba0cf2717a37a7baf5669028af47a44 WatchSource:0}: Error finding container 3b4b436048fa83ec8ef58cefd04102eb4ba0cf2717a37a7baf5669028af47a44: Status 404 returned error can't find the container with id 3b4b436048fa83ec8ef58cefd04102eb4ba0cf2717a37a7baf5669028af47a44
Sep 09 20:16:56.633619 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.633589    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-p5rkt"]
Sep 09 20:16:56.642962 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:56.642937    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749e9730_db41_43f8_914b_a435c9d671ff.slice/crio-2b0717c9f66b29da8dac3df97a2440f615f6ca7aa2d67cb1ecdfe6fe88178537 WatchSource:0}: Error finding container 2b0717c9f66b29da8dac3df97a2440f615f6ca7aa2d67cb1ecdfe6fe88178537: Status 404 returned error can't find the container with id 2b0717c9f66b29da8dac3df97a2440f615f6ca7aa2d67cb1ecdfe6fe88178537
Sep 09 20:16:56.646814 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.646791    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-c57sd"]
Sep 09 20:16:56.649806 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:56.649780    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be67041_c131_4335_a192_7476a5fcdbf1.slice/crio-725182f1847896b351eeb2070bae92e8be00e3494b39370aa3c36c90ffa7ed03 WatchSource:0}: Error finding container 725182f1847896b351eeb2070bae92e8be00e3494b39370aa3c36c90ffa7ed03: Status 404 returned error can't find the container with id 725182f1847896b351eeb2070bae92e8be00e3494b39370aa3c36c90ffa7ed03
Sep 09 20:16:56.663307 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.663282    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t552h"]
Sep 09 20:16:56.665913 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:16:56.665890    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab7b49e_a152_44f0_8ebb_3643d8862b0c.slice/crio-5bf752760dbf6293c357e8dc94f6fca37a76d9580a1ce4ea5f027dbd4894c8ae WatchSource:0}: Error finding container 5bf752760dbf6293c357e8dc94f6fca37a76d9580a1ce4ea5f027dbd4894c8ae: Status 404 returned error can't find the container with id 5bf752760dbf6293c357e8dc94f6fca37a76d9580a1ce4ea5f027dbd4894c8ae
Sep 09 20:16:56.700837 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.700805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.700978 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.700844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:56.700978 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.700945    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:16:56.700978 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.700945    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:16:56.700978 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.700961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:56.701201 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.701006    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls podName:48d0e306-fdd9-4e95-ab7c-5ebdbb06f444 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:57.700980518 +0000 UTC m=+39.194191498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls") pod "dns-default-z7wr5" (UID: "48d0e306-fdd9-4e95-ab7c-5ebdbb06f444") : secret "dns-default-metrics-tls" not found
Sep 09 20:16:56.701201 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.701050    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:57.701038663 +0000 UTC m=+39.194249625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : configmap references non-existent config key: service-ca.crt
Sep 09 20:16:56.701201 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:56.701116    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:57.701109585 +0000 UTC m=+39.194320546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : secret "router-metrics-certs-default" not found
Sep 09 20:16:56.996662 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.996622    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:16:56.999459 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.999430    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:16:56.999459 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:56.999441    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wv6m4\""
Sep 09 20:16:57.171196 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.171160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-c57sd" event={"ID":"2be67041-c131-4335-a192-7476a5fcdbf1","Type":"ContainerStarted","Data":"725182f1847896b351eeb2070bae92e8be00e3494b39370aa3c36c90ffa7ed03"}
Sep 09 20:16:57.172541 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.172514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt" event={"ID":"670f84b1-5f23-44cf-9b05-e4a3e603b7fa","Type":"ContainerStarted","Data":"f178af0b9870361dbf1b674ce6c6a5a8f4ccac866d089702b143c27c86623658"}
Sep 09 20:16:57.173941 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.173895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t552h" event={"ID":"dab7b49e-a152-44f0-8ebb-3643d8862b0c","Type":"ContainerStarted","Data":"5bf752760dbf6293c357e8dc94f6fca37a76d9580a1ce4ea5f027dbd4894c8ae"}
Sep 09 20:16:57.175278 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.175255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-7c664" event={"ID":"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68","Type":"ContainerStarted","Data":"d4a0aca08d88270d554fb0eacb3e1a58b37ddc255c84f9dca77ece14782b4216"}
Sep 09 20:16:57.176394 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.176373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-p5rkt" event={"ID":"749e9730-db41-43f8-914b-a435c9d671ff","Type":"ContainerStarted","Data":"2b0717c9f66b29da8dac3df97a2440f615f6ca7aa2d67cb1ecdfe6fe88178537"}
Sep 09 20:16:57.177560 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.177526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9" event={"ID":"83a81b0e-8d00-470d-b1b3-28b2832b2746","Type":"ContainerStarted","Data":"eaa70f4a9ae35bf4efb1fba8bcba429f63652774ebf7c6282b096b4f458c106c"}
Sep 09 20:16:57.178630 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.178607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b" event={"ID":"a32e9cee-8743-486e-a042-41ae301e7f11","Type":"ContainerStarted","Data":"3b4b436048fa83ec8ef58cefd04102eb4ba0cf2717a37a7baf5669028af47a44"}
Sep 09 20:16:57.609477 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.609434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:16:57.609639 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.609507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:57.609639 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.609612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:57.609808 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.609792    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:16:57.609857 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.609813    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9984f7d-8f8lz: secret "image-registry-tls" not found
Sep 09 20:16:57.609905 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.609895    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls podName:306d365e-c1ae-4944-868f-844b11fd59f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:59.60987499 +0000 UTC m=+41.103085966 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls") pod "image-registry-84f9984f7d-8f8lz" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4") : secret "image-registry-tls" not found
Sep 09 20:16:57.610528 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.610510    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:16:57.610612 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.610571    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls podName:bad2e819-d7a0-4f3a-bbd0-57b076806090 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:59.610554593 +0000 UTC m=+41.103765570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cwptv" (UID: "bad2e819-d7a0-4f3a-bbd0-57b076806090") : secret "samples-operator-tls" not found
Sep 09 20:16:57.610673 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.610629    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:16:57.610673 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.610663    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls podName:cc4d0a36-bc84-4ffe-b5d8-804d35114373 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:59.610651542 +0000 UTC m=+41.103862518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-mfr42" (UID: "cc4d0a36-bc84-4ffe-b5d8-804d35114373") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:16:57.711039 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.710992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:57.711204 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.711069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:57.711204 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:57.711114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:57.711412 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.711397    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:59.711375988 +0000 UTC m=+41.204586956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : configmap references non-existent config key: service-ca.crt
Sep 09 20:16:57.711910 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.711888    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:16:57.711985 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.711945    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:59.711929032 +0000 UTC m=+41.205140000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : secret "router-metrics-certs-default" not found
Sep 09 20:16:57.712107 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.712090    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:16:57.712167 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:57.712135    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls podName:48d0e306-fdd9-4e95-ab7c-5ebdbb06f444 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:59.712122946 +0000 UTC m=+41.205333922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls") pod "dns-default-z7wr5" (UID: "48d0e306-fdd9-4e95-ab7c-5ebdbb06f444") : secret "dns-default-metrics-tls" not found
Sep 09 20:16:58.891828 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:58.891459    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-mlwpc"]
Sep 09 20:16:58.908021 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:58.907971    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-mlwpc"]
Sep 09 20:16:58.908187 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:58.908089    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:16:58.914631 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:58.914607    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-r44qg\""
Sep 09 20:16:58.921911 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:58.914686    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 20:16:58.922067 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:58.914739    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 20:16:59.026941 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.026906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:16:59.027143 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.027024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e937672f-c395-458f-890d-aea135850301-nginx-conf\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:16:59.127980 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.127921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:16:59.128135 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.128048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e937672f-c395-458f-890d-aea135850301-nginx-conf\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:16:59.128944 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.128918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e937672f-c395-458f-890d-aea135850301-nginx-conf\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:16:59.129107 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.129088    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:16:59.129178 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.129161    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert podName:e937672f-c395-458f-890d-aea135850301 nodeName:}" failed. No retries permitted until 2026-09-09 20:16:59.629139998 +0000 UTC m=+41.122350976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-mlwpc" (UID: "e937672f-c395-458f-890d-aea135850301") : secret "networking-console-plugin-cert" not found
Sep 09 20:16:59.633811 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.633771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:16:59.634006 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.633869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:16:59.634006 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.633908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:16:59.634134 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.633947    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:16:59.634134 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.634013    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:16:59.634134 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.634048    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:16:59.634134 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.633978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:16:59.634134 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.634083    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:16:59.634134 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.634107    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9984f7d-8f8lz: secret "image-registry-tls" not found
Sep 09 20:16:59.634419 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.634145    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls podName:cc4d0a36-bc84-4ffe-b5d8-804d35114373 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:03.634059604 +0000 UTC m=+45.127270579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-mfr42" (UID: "cc4d0a36-bc84-4ffe-b5d8-804d35114373") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:16:59.634419 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.634174    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert podName:e937672f-c395-458f-890d-aea135850301 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:00.634162451 +0000 UTC m=+42.127373427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-mlwpc" (UID: "e937672f-c395-458f-890d-aea135850301") : secret "networking-console-plugin-cert" not found
Sep 09 20:16:59.634419 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.634194    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls podName:bad2e819-d7a0-4f3a-bbd0-57b076806090 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:03.634184321 +0000 UTC m=+45.127395287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cwptv" (UID: "bad2e819-d7a0-4f3a-bbd0-57b076806090") : secret "samples-operator-tls" not found
Sep 09 20:16:59.634419 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.634210    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls podName:306d365e-c1ae-4944-868f-844b11fd59f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:03.634202119 +0000 UTC m=+45.127413082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls") pod "image-registry-84f9984f7d-8f8lz" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4") : secret "image-registry-tls" not found
Sep 09 20:16:59.735180 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.735138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:59.735180 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.735177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:16:59.735456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:16:59.735208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:16:59.735456 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.735278    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:16:59.735456 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.735356    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:03.735323594 +0000 UTC m=+45.228534556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : secret "router-metrics-certs-default" not found
Sep 09 20:16:59.735456 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.735370    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:16:59.735456 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.735376    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:03.735367729 +0000 UTC m=+45.228578693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : configmap references non-existent config key: service-ca.crt
Sep 09 20:16:59.735456 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:16:59.735430    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls podName:48d0e306-fdd9-4e95-ab7c-5ebdbb06f444 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:03.735412874 +0000 UTC m=+45.228623860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls") pod "dns-default-z7wr5" (UID: "48d0e306-fdd9-4e95-ab7c-5ebdbb06f444") : secret "dns-default-metrics-tls" not found
Sep 09 20:17:00.643822 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:00.643787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:17:00.644365 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:00.643956    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:17:00.644365 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:00.644027    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert podName:e937672f-c395-458f-890d-aea135850301 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:02.64400934 +0000 UTC m=+44.137220318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-mlwpc" (UID: "e937672f-c395-458f-890d-aea135850301") : secret "networking-console-plugin-cert" not found
Sep 09 20:17:02.664444 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:02.664396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:17:02.664917 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:02.664534    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:17:02.664917 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:02.664629    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert podName:e937672f-c395-458f-890d-aea135850301 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:06.664610072 +0000 UTC m=+48.157821056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-mlwpc" (UID: "e937672f-c395-458f-890d-aea135850301") : secret "networking-console-plugin-cert" not found
Sep 09 20:17:03.672694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:03.672598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:17:03.672694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:03.672662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:17:03.673188 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.672756    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:17:03.673188 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:03.672769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:17:03.673188 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.672829    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:17:03.673188 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.672854    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls podName:bad2e819-d7a0-4f3a-bbd0-57b076806090 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:11.672832602 +0000 UTC m=+53.166043572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cwptv" (UID: "bad2e819-d7a0-4f3a-bbd0-57b076806090") : secret "samples-operator-tls" not found
Sep 09 20:17:03.673188 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.672888    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls podName:cc4d0a36-bc84-4ffe-b5d8-804d35114373 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:11.672870564 +0000 UTC m=+53.166081528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-mfr42" (UID: "cc4d0a36-bc84-4ffe-b5d8-804d35114373") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:17:03.673188 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.672976    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:17:03.673188 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.672998    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9984f7d-8f8lz: secret "image-registry-tls" not found
Sep 09 20:17:03.673188 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.673044    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls podName:306d365e-c1ae-4944-868f-844b11fd59f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:11.673032082 +0000 UTC m=+53.166243061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls") pod "image-registry-84f9984f7d-8f8lz" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4") : secret "image-registry-tls" not found
Sep 09 20:17:03.773803 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:03.773766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:03.773803 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:03.773818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:17:03.774046 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:03.773860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:03.774135 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.774114    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:11.774094418 +0000 UTC m=+53.267305407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : configmap references non-existent config key: service-ca.crt
Sep 09 20:17:03.774285 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.774215    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:17:03.774285 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.774274    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:11.774258036 +0000 UTC m=+53.267468998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : secret "router-metrics-certs-default" not found
Sep 09 20:17:03.774285 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.774215    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:17:03.774482 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:03.774310    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls podName:48d0e306-fdd9-4e95-ab7c-5ebdbb06f444 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:11.774301479 +0000 UTC m=+53.267512441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls") pod "dns-default-z7wr5" (UID: "48d0e306-fdd9-4e95-ab7c-5ebdbb06f444") : secret "dns-default-metrics-tls" not found
Sep 09 20:17:05.206036 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.205631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-c57sd" event={"ID":"2be67041-c131-4335-a192-7476a5fcdbf1","Type":"ContainerStarted","Data":"c26d55d7d105ca02d0f19f401c77a9eb360ae61e057be023f37f5280bb1aa1d7"}
Sep 09 20:17:05.207368 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.207327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt" event={"ID":"670f84b1-5f23-44cf-9b05-e4a3e603b7fa","Type":"ContainerStarted","Data":"6527fd4f50f61f4c985486e9a12252fa05e5b0c43e6c872ba64283751411819d"}
Sep 09 20:17:05.209141 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.209118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t552h" event={"ID":"dab7b49e-a152-44f0-8ebb-3643d8862b0c","Type":"ContainerStarted","Data":"5107e8d3cd534995b3575d07871d1ba7a1193ec92eccf90df7a59f718327c7f0"}
Sep 09 20:17:05.210851 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.210831    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/0.log"
Sep 09 20:17:05.210959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.210874    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68" containerID="f29fd7db5cfbeb27a284b1fd2731298a50173abcc82019c6b7d701b6ec3a1ce4" exitCode=255
Sep 09 20:17:05.210959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.210949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-7c664" event={"ID":"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68","Type":"ContainerDied","Data":"f29fd7db5cfbeb27a284b1fd2731298a50173abcc82019c6b7d701b6ec3a1ce4"}
Sep 09 20:17:05.211158 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.211140    2581 scope.go:117] "RemoveContainer" containerID="f29fd7db5cfbeb27a284b1fd2731298a50173abcc82019c6b7d701b6ec3a1ce4"
Sep 09 20:17:05.215059 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.215036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-p5rkt" event={"ID":"749e9730-db41-43f8-914b-a435c9d671ff","Type":"ContainerStarted","Data":"0d05676bfafea31738c9ad8214c1150c49fcefca57ea4bfae4756640aa949fd6"}
Sep 09 20:17:05.216718 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.216694    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9" event={"ID":"83a81b0e-8d00-470d-b1b3-28b2832b2746","Type":"ContainerStarted","Data":"d7b5b5d6e2a0cd0c1c487ae6820820e396b17b037fa7fd8c9baad8ddb021f282"}
Sep 09 20:17:05.218528 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.218506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b" event={"ID":"a32e9cee-8743-486e-a042-41ae301e7f11","Type":"ContainerStarted","Data":"b6c358e8d21302d62c2f5d22bbf29dcab22f446ebba0c832ba0e25e9abc73f3a"}
Sep 09 20:17:05.224387 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.223831    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-c57sd" podStartSLOduration=10.469605646 podStartE2EDuration="18.22381613s" totalImagesPullingTime="7.754210484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:05.223168626 +0000 UTC m=+46.716379603" watchObservedRunningTime="2026-09-09 20:17:05.22381613 +0000 UTC m=+46.717027110"
Sep 09 20:17:05.246613 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.245489    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b" podStartSLOduration=10.411508051 podStartE2EDuration="18.245468964s" totalImagesPullingTime="7.833960913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:05.243252496 +0000 UTC m=+46.736463478" watchObservedRunningTime="2026-09-09 20:17:05.245468964 +0000 UTC m=+46.738679949"
Sep 09 20:17:05.290326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.289820    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt" podStartSLOduration=10.420507199 podStartE2EDuration="18.289797849s" totalImagesPullingTime="7.86929065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:05.289519339 +0000 UTC m=+46.782730324" watchObservedRunningTime="2026-09-09 20:17:05.289797849 +0000 UTC m=+46.783008834"
Sep 09 20:17:05.313493 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.313414    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9" podStartSLOduration=10.479520371 podStartE2EDuration="18.313394466s" totalImagesPullingTime="7.833874095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:05.313004477 +0000 UTC m=+46.806215462" watchObservedRunningTime="2026-09-09 20:17:05.313394466 +0000 UTC m=+46.806605452"
Sep 09 20:17:05.330223 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.330168    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-p5rkt" podStartSLOduration=10.393253501 podStartE2EDuration="18.330148056s" totalImagesPullingTime="7.936894555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:05.328953148 +0000 UTC m=+46.822164133" watchObservedRunningTime="2026-09-09 20:17:05.330148056 +0000 UTC m=+46.823359043"
Sep 09 20:17:05.349324 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:05.347913    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t552h" podStartSLOduration=2.579914975 podStartE2EDuration="10.347893651s" totalImagesPullingTime="7.767978676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:05.347472253 +0000 UTC m=+46.840683239" watchObservedRunningTime="2026-09-09 20:17:05.347893651 +0000 UTC m=+46.841104637"
Sep 09 20:17:06.014480 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.014450    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sgqkp"]
Sep 09 20:17:06.027747 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.027724    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.029473 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.029441    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sgqkp"]
Sep 09 20:17:06.030720 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.030697    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:17:06.030845 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.030704    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:17:06.031964 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.031943    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s82pj\""
Sep 09 20:17:06.185191 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.185158    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:17:06.185333 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.185212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:17:06.197533 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.197503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbn2h\" (UniqueName: \"kubernetes.io/projected/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-api-access-wbn2h\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.197622 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.197558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aaad6fa0-2376-4fc7-976a-395d763899dc-crio-socket\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.197773 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.197752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.197816 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.197794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.197854 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.197833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aaad6fa0-2376-4fc7-976a-395d763899dc-data-volume\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.224516 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.224493    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/1.log"
Sep 09 20:17:06.224970 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.224955    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/0.log"
Sep 09 20:17:06.225027 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.224999    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68" containerID="bb2ac7778a2fc7c20865020b7c8387bf95a08f1e7fc5d32f08223569bb68dd39" exitCode=255
Sep 09 20:17:06.225093 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.225075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-7c664" event={"ID":"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68","Type":"ContainerDied","Data":"bb2ac7778a2fc7c20865020b7c8387bf95a08f1e7fc5d32f08223569bb68dd39"}
Sep 09 20:17:06.225139 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.225121    2581 scope.go:117] "RemoveContainer" containerID="f29fd7db5cfbeb27a284b1fd2731298a50173abcc82019c6b7d701b6ec3a1ce4"
Sep 09 20:17:06.225315 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.225300    2581 scope.go:117] "RemoveContainer" containerID="bb2ac7778a2fc7c20865020b7c8387bf95a08f1e7fc5d32f08223569bb68dd39"
Sep 09 20:17:06.225521 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:06.225498    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-7c664_openshift-console-operator(e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68)\"" pod="openshift-console-operator/console-operator-85b454874b-7c664" podUID="e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68"
Sep 09 20:17:06.299394 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.299294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbn2h\" (UniqueName: \"kubernetes.io/projected/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-api-access-wbn2h\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.299394 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.299387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aaad6fa0-2376-4fc7-976a-395d763899dc-crio-socket\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.299603 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.299475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.299603 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.299511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.299603 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.299536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aaad6fa0-2376-4fc7-976a-395d763899dc-data-volume\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.299603 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:06.299594    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:06.299739 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.299644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aaad6fa0-2376-4fc7-976a-395d763899dc-crio-socket\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.299739 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:06.299653    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:17:06.299798 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:06.299744    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm podName:aaad6fa0-2376-4fc7-976a-395d763899dc nodeName:}" failed. No retries permitted until 2026-09-09 20:17:06.799725014 +0000 UTC m=+48.292935977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sgqkp" (UID: "aaad6fa0-2376-4fc7-976a-395d763899dc") : configmap "kube-rbac-proxy" not found
Sep 09 20:17:06.299917 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:06.299902    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls podName:aaad6fa0-2376-4fc7-976a-395d763899dc nodeName:}" failed. No retries permitted until 2026-09-09 20:17:06.799881893 +0000 UTC m=+48.293092856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sgqkp" (UID: "aaad6fa0-2376-4fc7-976a-395d763899dc") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:06.299969 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.299940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aaad6fa0-2376-4fc7-976a-395d763899dc-data-volume\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.316701 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.316676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbn2h\" (UniqueName: \"kubernetes.io/projected/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-api-access-wbn2h\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.704813 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.704779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:17:06.704962 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:06.704932    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:17:06.705023 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:06.705007    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert podName:e937672f-c395-458f-890d-aea135850301 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:14.704982897 +0000 UTC m=+56.198193859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-mlwpc" (UID: "e937672f-c395-458f-890d-aea135850301") : secret "networking-console-plugin-cert" not found
Sep 09 20:17:06.805651 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.805615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.805651 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.805657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:06.805877 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:06.805792    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:17:06.805877 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:06.805789    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:06.805877 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:06.805850    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm podName:aaad6fa0-2376-4fc7-976a-395d763899dc nodeName:}" failed. No retries permitted until 2026-09-09 20:17:07.805833664 +0000 UTC m=+49.299044627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sgqkp" (UID: "aaad6fa0-2376-4fc7-976a-395d763899dc") : configmap "kube-rbac-proxy" not found
Sep 09 20:17:06.805877 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:06.805863    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls podName:aaad6fa0-2376-4fc7-976a-395d763899dc nodeName:}" failed. No retries permitted until 2026-09-09 20:17:07.805857112 +0000 UTC m=+49.299068074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sgqkp" (UID: "aaad6fa0-2376-4fc7-976a-395d763899dc") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:06.949717 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:06.949693    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2xswt_e5088b6b-85b2-4c2f-b7b6-0f1039ba191a/dns-node-resolver/0.log"
Sep 09 20:17:07.232635 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.232611    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/1.log"
Sep 09 20:17:07.233073 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.232908    2581 scope.go:117] "RemoveContainer" containerID="bb2ac7778a2fc7c20865020b7c8387bf95a08f1e7fc5d32f08223569bb68dd39"
Sep 09 20:17:07.233073 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:07.233064    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-7c664_openshift-console-operator(e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68)\"" pod="openshift-console-operator/console-operator-85b454874b-7c664" podUID="e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68"
Sep 09 20:17:07.329254 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.329185    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49"]
Sep 09 20:17:07.365617 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.365591    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49"]
Sep 09 20:17:07.365760 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.365678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49"
Sep 09 20:17:07.368248 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.368229    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 20:17:07.368444 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.368427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-whdkq\""
Sep 09 20:17:07.368735 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.368720    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 20:17:07.410949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.410927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5q9\" (UniqueName: \"kubernetes.io/projected/f51200f4-121b-42d4-bae7-d4609abc374d-kube-api-access-wr5q9\") pod \"migrator-fdbb8fdff-2kw49\" (UID: \"f51200f4-121b-42d4-bae7-d4609abc374d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49"
Sep 09 20:17:07.511924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.511894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5q9\" (UniqueName: \"kubernetes.io/projected/f51200f4-121b-42d4-bae7-d4609abc374d-kube-api-access-wr5q9\") pod \"migrator-fdbb8fdff-2kw49\" (UID: \"f51200f4-121b-42d4-bae7-d4609abc374d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49"
Sep 09 20:17:07.525311 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.525276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5q9\" (UniqueName: \"kubernetes.io/projected/f51200f4-121b-42d4-bae7-d4609abc374d-kube-api-access-wr5q9\") pod \"migrator-fdbb8fdff-2kw49\" (UID: \"f51200f4-121b-42d4-bae7-d4609abc374d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49"
Sep 09 20:17:07.675058 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.674973    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49"
Sep 09 20:17:07.814295 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.814263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:07.814457 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.814309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:07.814457 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:07.814447    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:07.814546 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:07.814473    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:17:07.814546 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:07.814519    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls podName:aaad6fa0-2376-4fc7-976a-395d763899dc nodeName:}" failed. No retries permitted until 2026-09-09 20:17:09.81450151 +0000 UTC m=+51.307712476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sgqkp" (UID: "aaad6fa0-2376-4fc7-976a-395d763899dc") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:07.814546 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:07.814534    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm podName:aaad6fa0-2376-4fc7-976a-395d763899dc nodeName:}" failed. No retries permitted until 2026-09-09 20:17:09.814527884 +0000 UTC m=+51.307738846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sgqkp" (UID: "aaad6fa0-2376-4fc7-976a-395d763899dc") : configmap "kube-rbac-proxy" not found
Sep 09 20:17:07.949610 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.949589    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2859f_83456891-bf2f-4d7d-82a6-cb2820c78575/node-ca/0.log"
Sep 09 20:17:07.966764 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:07.966736    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49"]
Sep 09 20:17:07.969739 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:17:07.969705    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51200f4_121b_42d4_bae7_d4609abc374d.slice/crio-0ec62040808899f81319408245af6163fdad18e3410a963dc55d89e74b426bb2 WatchSource:0}: Error finding container 0ec62040808899f81319408245af6163fdad18e3410a963dc55d89e74b426bb2: Status 404 returned error can't find the container with id 0ec62040808899f81319408245af6163fdad18e3410a963dc55d89e74b426bb2
Sep 09 20:17:08.237005 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:08.236916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49" event={"ID":"f51200f4-121b-42d4-bae7-d4609abc374d","Type":"ContainerStarted","Data":"0ec62040808899f81319408245af6163fdad18e3410a963dc55d89e74b426bb2"}
Sep 09 20:17:08.750029 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:08.749995    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t552h_dab7b49e-a152-44f0-8ebb-3643d8862b0c/serve-healthcheck-canary/0.log"
Sep 09 20:17:09.831294 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:09.831266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:09.831588 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:09.831324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:09.831588 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:09.831391    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:09.831588 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:09.831432    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:17:09.831588 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:09.831459    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls podName:aaad6fa0-2376-4fc7-976a-395d763899dc nodeName:}" failed. No retries permitted until 2026-09-09 20:17:13.83144079 +0000 UTC m=+55.324651776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sgqkp" (UID: "aaad6fa0-2376-4fc7-976a-395d763899dc") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:09.831588 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:09.831504    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm podName:aaad6fa0-2376-4fc7-976a-395d763899dc nodeName:}" failed. No retries permitted until 2026-09-09 20:17:13.831497495 +0000 UTC m=+55.324708457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sgqkp" (UID: "aaad6fa0-2376-4fc7-976a-395d763899dc") : configmap "kube-rbac-proxy" not found
Sep 09 20:17:10.245215 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:10.245180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49" event={"ID":"f51200f4-121b-42d4-bae7-d4609abc374d","Type":"ContainerStarted","Data":"fd31b474bd22b6ce06c72b17ae12981897ec6f838d7d6705af7637bd57843e0f"}
Sep 09 20:17:10.245409 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:10.245262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49" event={"ID":"f51200f4-121b-42d4-bae7-d4609abc374d","Type":"ContainerStarted","Data":"62c265e4c832c9bee7196567faa4c40609efbd24c220d4c299b70ac8f13299cc"}
Sep 09 20:17:10.267803 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:10.267759    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2kw49" podStartSLOduration=1.643972188 podStartE2EDuration="3.267740233s" totalImagesPullingTime="1.623768045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:10.265861245 +0000 UTC m=+51.759072229" watchObservedRunningTime="2026-09-09 20:17:10.267740233 +0000 UTC m=+51.760951251"
Sep 09 20:17:11.748199 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:11.748155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:17:11.748802 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:11.748209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:17:11.748802 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.748306    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 20:17:11.748802 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:11.748325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:17:11.748802 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.748389    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:17:11.748802 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.748397    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls podName:bad2e819-d7a0-4f3a-bbd0-57b076806090 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:27.748375442 +0000 UTC m=+69.241586404 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-cwptv" (UID: "bad2e819-d7a0-4f3a-bbd0-57b076806090") : secret "samples-operator-tls" not found
Sep 09 20:17:11.748802 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.748452    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls podName:cc4d0a36-bc84-4ffe-b5d8-804d35114373 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:27.748437488 +0000 UTC m=+69.241648450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-mfr42" (UID: "cc4d0a36-bc84-4ffe-b5d8-804d35114373") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:17:11.748802 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.748454    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:17:11.748802 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.748471    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84f9984f7d-8f8lz: secret "image-registry-tls" not found
Sep 09 20:17:11.748802 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.748505    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls podName:306d365e-c1ae-4944-868f-844b11fd59f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:27.748495092 +0000 UTC m=+69.241706054 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls") pod "image-registry-84f9984f7d-8f8lz" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4") : secret "image-registry-tls" not found
Sep 09 20:17:11.849015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:11.848979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:11.849197 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:11.849022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:17:11.849197 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.849114    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:17:11.849197 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.849117    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 20:17:11.849197 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.849164    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls podName:48d0e306-fdd9-4e95-ab7c-5ebdbb06f444 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:27.849149636 +0000 UTC m=+69.342360615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls") pod "dns-default-z7wr5" (UID: "48d0e306-fdd9-4e95-ab7c-5ebdbb06f444") : secret "dns-default-metrics-tls" not found
Sep 09 20:17:11.849197 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.849183    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:27.849170707 +0000 UTC m=+69.342381668 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : secret "router-metrics-certs-default" not found
Sep 09 20:17:11.849197 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:11.849155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:11.849510 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:11.849274    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle podName:23891ba7-4710-4468-b69d-8764b3cf8384 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:27.849258635 +0000 UTC m=+69.342469613 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle") pod "router-default-5c4444dc76-c2dcn" (UID: "23891ba7-4710-4468-b69d-8764b3cf8384") : configmap references non-existent config key: service-ca.crt
Sep 09 20:17:13.864317 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:13.864278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:13.864317 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:13.864317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:13.864819 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:13.864436    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:17:13.864819 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:13.864478    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:13.864819 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:13.864489    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm podName:aaad6fa0-2376-4fc7-976a-395d763899dc nodeName:}" failed. No retries permitted until 2026-09-09 20:17:21.864476357 +0000 UTC m=+63.357687319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sgqkp" (UID: "aaad6fa0-2376-4fc7-976a-395d763899dc") : configmap "kube-rbac-proxy" not found
Sep 09 20:17:13.864819 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:13.864560    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls podName:aaad6fa0-2376-4fc7-976a-395d763899dc nodeName:}" failed. No retries permitted until 2026-09-09 20:17:21.86454086 +0000 UTC m=+63.357751822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sgqkp" (UID: "aaad6fa0-2376-4fc7-976a-395d763899dc") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:14.773139 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:14.773101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:17:14.773332 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:14.773248    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 20:17:14.773332 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:14.773312    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert podName:e937672f-c395-458f-890d-aea135850301 nodeName:}" failed. No retries permitted until 2026-09-09 20:17:30.773296792 +0000 UTC m=+72.266507754 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-mlwpc" (UID: "e937672f-c395-458f-890d-aea135850301") : secret "networking-console-plugin-cert" not found
Sep 09 20:17:16.185164 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:16.185129    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:17:16.185614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:16.185222    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:17:16.185614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:16.185607    2581 scope.go:117] "RemoveContainer" containerID="bb2ac7778a2fc7c20865020b7c8387bf95a08f1e7fc5d32f08223569bb68dd39"
Sep 09 20:17:16.265885 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:16.265864    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/1.log"
Sep 09 20:17:16.265998 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:16.265924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-7c664" event={"ID":"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68","Type":"ContainerStarted","Data":"24a7bb8b971a1f8daa8297d9832dc4da7f92cf5f821dc410ba2ba85a1bca7b13"}
Sep 09 20:17:16.266194 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:16.266176    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:17:16.267674 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:16.267640    2581 patch_prober.go:28] interesting pod/console-operator-85b454874b-7c664 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused" start-of-body=
Sep 09 20:17:16.267783 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:16.267688    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-85b454874b-7c664" podUID="e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.8:8443/readyz\": dial tcp 10.132.0.8:8443: connect: connection refused"
Sep 09 20:17:16.284656 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:16.284602    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-7c664" podStartSLOduration=21.362385593 podStartE2EDuration="29.284582619s" totalImagesPullingTime="7.922197026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:16.283784351 +0000 UTC m=+57.776995335" watchObservedRunningTime="2026-09-09 20:17:16.284582619 +0000 UTC m=+57.777793604"
Sep 09 20:17:17.272646 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:17.272620    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:17:17.273156 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:17.273141    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/1.log"
Sep 09 20:17:17.273211 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:17.273180    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68" containerID="24a7bb8b971a1f8daa8297d9832dc4da7f92cf5f821dc410ba2ba85a1bca7b13" exitCode=255
Sep 09 20:17:17.273272 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:17.273222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-7c664" event={"ID":"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68","Type":"ContainerDied","Data":"24a7bb8b971a1f8daa8297d9832dc4da7f92cf5f821dc410ba2ba85a1bca7b13"}
Sep 09 20:17:17.273308 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:17.273277    2581 scope.go:117] "RemoveContainer" containerID="bb2ac7778a2fc7c20865020b7c8387bf95a08f1e7fc5d32f08223569bb68dd39"
Sep 09 20:17:17.273542 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:17.273514    2581 scope.go:117] "RemoveContainer" containerID="24a7bb8b971a1f8daa8297d9832dc4da7f92cf5f821dc410ba2ba85a1bca7b13"
Sep 09 20:17:17.273703 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:17.273684    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-7c664_openshift-console-operator(e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68)\"" pod="openshift-console-operator/console-operator-85b454874b-7c664" podUID="e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68"
Sep 09 20:17:18.278576 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:18.278552    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:17:18.278965 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:18.278906    2581 scope.go:117] "RemoveContainer" containerID="24a7bb8b971a1f8daa8297d9832dc4da7f92cf5f821dc410ba2ba85a1bca7b13"
Sep 09 20:17:18.279084 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:18.279065    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-7c664_openshift-console-operator(e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68)\"" pod="openshift-console-operator/console-operator-85b454874b-7c664" podUID="e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68"
Sep 09 20:17:19.152556 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:19.152530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kpdtc"
Sep 09 20:17:21.938487 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:21.938452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:21.938487 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:21.938490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:21.938875 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:21.938605    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:21.938875 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:21.938694    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls podName:aaad6fa0-2376-4fc7-976a-395d763899dc nodeName:}" failed. No retries permitted until 2026-09-09 20:17:37.938678043 +0000 UTC m=+79.431889022 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sgqkp" (UID: "aaad6fa0-2376-4fc7-976a-395d763899dc") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:17:21.939014 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:21.938993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aaad6fa0-2376-4fc7-976a-395d763899dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:24.761922 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.761856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4klm\" (UniqueName: \"kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm\") pod \"network-check-target-lf5rw\" (UID: \"9e614f95-303b-473d-b59b-544cc9a5ce53\") " pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:17:24.764562 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.764543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4klm\" (UniqueName: \"kubernetes.io/projected/9e614f95-303b-473d-b59b-544cc9a5ce53-kube-api-access-s4klm\") pod \"network-check-target-lf5rw\" (UID: \"9e614f95-303b-473d-b59b-544cc9a5ce53\") " pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:17:24.828594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.828558    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c88hn\""
Sep 09 20:17:24.835833 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.835807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:17:24.863478 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.863447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:17:24.863646 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.863512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:17:24.866540 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.866522    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:17:24.866596 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.866522    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:17:24.876245 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.876218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3774603a-6336-4e25-82a3-c5a0b89fd082-original-pull-secret\") pod \"global-pull-secret-syncer-2f9sh\" (UID: \"3774603a-6336-4e25-82a3-c5a0b89fd082\") " pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:17:24.876377 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.876255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ae6a25-16b2-48d6-98dd-dcdc41d99f7e-metrics-certs\") pod \"network-metrics-daemon-rbt27\" (UID: \"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e\") " pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:17:24.908305 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.908001    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wv6m4\""
Sep 09 20:17:24.915883 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:24.915852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rbt27"
Sep 09 20:17:25.111572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:25.111497    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lf5rw"]
Sep 09 20:17:25.114246 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:17:25.114221    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e614f95_303b_473d_b59b_544cc9a5ce53.slice/crio-a3dc8cb232fd10ddb513cd0abbff7c8937fe2ae45afd6f62cb5a0b77cfff22ad WatchSource:0}: Error finding container a3dc8cb232fd10ddb513cd0abbff7c8937fe2ae45afd6f62cb5a0b77cfff22ad: Status 404 returned error can't find the container with id a3dc8cb232fd10ddb513cd0abbff7c8937fe2ae45afd6f62cb5a0b77cfff22ad
Sep 09 20:17:25.118754 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:25.118730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f9sh"
Sep 09 20:17:25.182896 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:25.182869    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rbt27"]
Sep 09 20:17:25.186033 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:17:25.186004    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ae6a25_16b2_48d6_98dd_dcdc41d99f7e.slice/crio-d28e312ebfe6cc8a54c42b8ff077588b8b5d1dc926a7f973868a4bdd0c01eb6e WatchSource:0}: Error finding container d28e312ebfe6cc8a54c42b8ff077588b8b5d1dc926a7f973868a4bdd0c01eb6e: Status 404 returned error can't find the container with id d28e312ebfe6cc8a54c42b8ff077588b8b5d1dc926a7f973868a4bdd0c01eb6e
Sep 09 20:17:25.299778 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:25.299739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lf5rw" event={"ID":"9e614f95-303b-473d-b59b-544cc9a5ce53","Type":"ContainerStarted","Data":"88ce9a3cdcc80ee5685215b17be6524f3e505f26218c31af589e545552084d8c"}
Sep 09 20:17:25.299973 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:25.299794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lf5rw" event={"ID":"9e614f95-303b-473d-b59b-544cc9a5ce53","Type":"ContainerStarted","Data":"a3dc8cb232fd10ddb513cd0abbff7c8937fe2ae45afd6f62cb5a0b77cfff22ad"}
Sep 09 20:17:25.299973 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:25.299852    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:17:25.301133 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:25.301109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rbt27" event={"ID":"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e","Type":"ContainerStarted","Data":"d28e312ebfe6cc8a54c42b8ff077588b8b5d1dc926a7f973868a4bdd0c01eb6e"}
Sep 09 20:17:25.319819 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:25.319778    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lf5rw" podStartSLOduration=66.319764747 podStartE2EDuration="1m6.319764747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:25.318527104 +0000 UTC m=+66.811738088" watchObservedRunningTime="2026-09-09 20:17:25.319764747 +0000 UTC m=+66.812975722"
Sep 09 20:17:25.393120 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:25.393049    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2f9sh"]
Sep 09 20:17:25.395654 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:17:25.395632    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3774603a_6336_4e25_82a3_c5a0b89fd082.slice/crio-b84dab4a4feea3d0f3aa83dd8339592b3984c273d28819a1bd9472feabb24260 WatchSource:0}: Error finding container b84dab4a4feea3d0f3aa83dd8339592b3984c273d28819a1bd9472feabb24260: Status 404 returned error can't find the container with id b84dab4a4feea3d0f3aa83dd8339592b3984c273d28819a1bd9472feabb24260
Sep 09 20:17:26.185535 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:26.185497    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:17:26.186000 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:26.185979    2581 scope.go:117] "RemoveContainer" containerID="24a7bb8b971a1f8daa8297d9832dc4da7f92cf5f821dc410ba2ba85a1bca7b13"
Sep 09 20:17:26.186201 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:17:26.186167    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-7c664_openshift-console-operator(e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68)\"" pod="openshift-console-operator/console-operator-85b454874b-7c664" podUID="e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68"
Sep 09 20:17:26.307272 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:26.307240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2f9sh" event={"ID":"3774603a-6336-4e25-82a3-c5a0b89fd082","Type":"ContainerStarted","Data":"b84dab4a4feea3d0f3aa83dd8339592b3984c273d28819a1bd9472feabb24260"}
Sep 09 20:17:27.312968 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.312922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rbt27" event={"ID":"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e","Type":"ContainerStarted","Data":"d6e6def963895e355a07371e4fea30d24613be0403540b0b1267edbee75a7feb"}
Sep 09 20:17:27.313435 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.312992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rbt27" event={"ID":"60ae6a25-16b2-48d6-98dd-dcdc41d99f7e","Type":"ContainerStarted","Data":"8d389bfdd3688870a9b9fcc9181c59f10f56aa2a4813ca23eb4aef0433187e8c"}
Sep 09 20:17:27.331687 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.331634    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rbt27" podStartSLOduration=67.267765598 podStartE2EDuration="1m8.33161876s" totalImagesPullingTime="1.063853162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:27.329484065 +0000 UTC m=+68.822695049" watchObservedRunningTime="2026-09-09 20:17:27.33161876 +0000 UTC m=+68.824829744"
Sep 09 20:17:27.791672 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.791629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:17:27.791927 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.791750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:17:27.791927 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.791795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:17:27.794365 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.794316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls\") pod \"image-registry-84f9984f7d-8f8lz\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") " pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:17:27.794529 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.794508    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc4d0a36-bc84-4ffe-b5d8-804d35114373-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-mfr42\" (UID: \"cc4d0a36-bc84-4ffe-b5d8-804d35114373\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:17:27.794596 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.794573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bad2e819-d7a0-4f3a-bbd0-57b076806090-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-cwptv\" (UID: \"bad2e819-d7a0-4f3a-bbd0-57b076806090\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:17:27.893424 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.893387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:27.893614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.893434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:17:27.893614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.893470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:27.894260 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.894236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23891ba7-4710-4468-b69d-8764b3cf8384-service-ca-bundle\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:27.896180 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.896153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/48d0e306-fdd9-4e95-ab7c-5ebdbb06f444-metrics-tls\") pod \"dns-default-z7wr5\" (UID: \"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444\") " pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:17:27.896180 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.896169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23891ba7-4710-4468-b69d-8764b3cf8384-metrics-certs\") pod \"router-default-5c4444dc76-c2dcn\" (UID: \"23891ba7-4710-4468-b69d-8764b3cf8384\") " pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:27.958961 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.958935    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qrtsj\""
Sep 09 20:17:27.967122 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.967096    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:17:27.968428 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.968406    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c7xds\""
Sep 09 20:17:27.978105 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:27.978025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"
Sep 09 20:17:28.015587 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:28.014370    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hxwhc\""
Sep 09 20:17:28.022699 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:28.022639    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"
Sep 09 20:17:28.100356 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:28.100292    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-blkcn\""
Sep 09 20:17:28.108524 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:28.108500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:28.132963 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:28.132939    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8pv6p\""
Sep 09 20:17:28.140598 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:28.140412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:17:28.259646 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:28.259567    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84f9984f7d-8f8lz"]
Sep 09 20:17:28.261313 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:28.261294    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv"]
Sep 09 20:17:28.306261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:28.306231    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42"]
Sep 09 20:17:28.388379 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:28.388273    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c4444dc76-c2dcn"]
Sep 09 20:17:28.417782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:28.417450    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z7wr5"]
Sep 09 20:17:29.624018 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:17:29.623971    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4d0a36_bc84_4ffe_b5d8_804d35114373.slice/crio-015d3a9d752cc239902b6c4e0ab45d9794cda64b81910e49e9b3c177ebf10a21 WatchSource:0}: Error finding container 015d3a9d752cc239902b6c4e0ab45d9794cda64b81910e49e9b3c177ebf10a21: Status 404 returned error can't find the container with id 015d3a9d752cc239902b6c4e0ab45d9794cda64b81910e49e9b3c177ebf10a21
Sep 09 20:17:29.624902 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:17:29.624593    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d0e306_fdd9_4e95_ab7c_5ebdbb06f444.slice/crio-f81c008d00bfccaa5ea1b0afd223de532b241bbefcd7e12c27de838e5b3955af WatchSource:0}: Error finding container f81c008d00bfccaa5ea1b0afd223de532b241bbefcd7e12c27de838e5b3955af: Status 404 returned error can't find the container with id f81c008d00bfccaa5ea1b0afd223de532b241bbefcd7e12c27de838e5b3955af
Sep 09 20:17:29.625646 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:17:29.625624    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23891ba7_4710_4468_b69d_8764b3cf8384.slice/crio-bf18531d86697c9697a65b674ccd85ce858d6f547bc20de2c6bab22382c2049d WatchSource:0}: Error finding container bf18531d86697c9697a65b674ccd85ce858d6f547bc20de2c6bab22382c2049d: Status 404 returned error can't find the container with id bf18531d86697c9697a65b674ccd85ce858d6f547bc20de2c6bab22382c2049d
Sep 09 20:17:29.626537 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:17:29.626463    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod306d365e_c1ae_4944_868f_844b11fd59f4.slice/crio-1241d4afeb674e77de7b405415f511340a54b6284329f0010375c4d8f19c8f4b WatchSource:0}: Error finding container 1241d4afeb674e77de7b405415f511340a54b6284329f0010375c4d8f19c8f4b: Status 404 returned error can't find the container with id 1241d4afeb674e77de7b405415f511340a54b6284329f0010375c4d8f19c8f4b
Sep 09 20:17:30.326332 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.326282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv" event={"ID":"bad2e819-d7a0-4f3a-bbd0-57b076806090","Type":"ContainerStarted","Data":"84d89e3be5e301a3333e7f71fba8e79b3e6d3d5aa6b595ea448e37cb455587db"}
Sep 09 20:17:30.328483 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.328454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2f9sh" event={"ID":"3774603a-6336-4e25-82a3-c5a0b89fd082","Type":"ContainerStarted","Data":"a7d772afe38fe3422b430c566b77f2d1b95b4d91f0ff31c0cafe222e5596f8cc"}
Sep 09 20:17:30.331777 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.331746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c4444dc76-c2dcn" event={"ID":"23891ba7-4710-4468-b69d-8764b3cf8384","Type":"ContainerStarted","Data":"dd24c2f18993a7ecdb106f685fe2d5481f7f9f9847aa84d094b9ea26b2ec523c"}
Sep 09 20:17:30.331880 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.331800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c4444dc76-c2dcn" event={"ID":"23891ba7-4710-4468-b69d-8764b3cf8384","Type":"ContainerStarted","Data":"bf18531d86697c9697a65b674ccd85ce858d6f547bc20de2c6bab22382c2049d"}
Sep 09 20:17:30.333758 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.333734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z7wr5" event={"ID":"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444","Type":"ContainerStarted","Data":"f81c008d00bfccaa5ea1b0afd223de532b241bbefcd7e12c27de838e5b3955af"}
Sep 09 20:17:30.335293 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.335247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42" event={"ID":"cc4d0a36-bc84-4ffe-b5d8-804d35114373","Type":"ContainerStarted","Data":"015d3a9d752cc239902b6c4e0ab45d9794cda64b81910e49e9b3c177ebf10a21"}
Sep 09 20:17:30.337109 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.337083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz" event={"ID":"306d365e-c1ae-4944-868f-844b11fd59f4","Type":"ContainerStarted","Data":"d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571"}
Sep 09 20:17:30.337208 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.337124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz" event={"ID":"306d365e-c1ae-4944-868f-844b11fd59f4","Type":"ContainerStarted","Data":"1241d4afeb674e77de7b405415f511340a54b6284329f0010375c4d8f19c8f4b"}
Sep 09 20:17:30.337809 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.337399    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:17:30.345940 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.345894    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2f9sh" podStartSLOduration=67.06052534 podStartE2EDuration="1m11.345880781s" totalImagesPullingTime="4.285355441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:30.345508619 +0000 UTC m=+71.838719645" watchObservedRunningTime="2026-09-09 20:17:30.345880781 +0000 UTC m=+71.839091765"
Sep 09 20:17:30.367787 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.367742    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz" podStartSLOduration=71.367729973 podStartE2EDuration="1m11.367729973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:30.367229864 +0000 UTC m=+71.860440848" watchObservedRunningTime="2026-09-09 20:17:30.367729973 +0000 UTC m=+71.860940956"
Sep 09 20:17:30.387842 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.387793    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5c4444dc76-c2dcn" podStartSLOduration=43.387774967 podStartE2EDuration="43.387774967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:30.387162129 +0000 UTC m=+71.880373115" watchObservedRunningTime="2026-09-09 20:17:30.387774967 +0000 UTC m=+71.880985953"
Sep 09 20:17:30.818337 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.818294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:17:30.821254 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:30.821228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e937672f-c395-458f-890d-aea135850301-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-mlwpc\" (UID: \"e937672f-c395-458f-890d-aea135850301\") " pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:17:31.029356 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:31.029304    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-r44qg\""
Sep 09 20:17:31.034501 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:31.034281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc"
Sep 09 20:17:31.108985 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:31.108914    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:31.111885 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:31.111861    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:31.341409 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:31.341380    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:31.342864 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:31.342837    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c4444dc76-c2dcn"
Sep 09 20:17:33.357909 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:33.357019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z7wr5" event={"ID":"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444","Type":"ContainerStarted","Data":"a8fce0215dfa5d250b7975395bda1a535bd46ff287f2b763dc258c40594bc8b7"}
Sep 09 20:17:33.358877 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:33.358849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42" event={"ID":"cc4d0a36-bc84-4ffe-b5d8-804d35114373","Type":"ContainerStarted","Data":"40bccc81cd9d8507a2cee611b0d83ea35e060fbfc5f6e62d4ab57b5508774e23"}
Sep 09 20:17:33.375926 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:33.375817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv" event={"ID":"bad2e819-d7a0-4f3a-bbd0-57b076806090","Type":"ContainerStarted","Data":"3db2c55c5a67be528d0ec04f9bc4be28c688584cc3da54a29337f9649b99639f"}
Sep 09 20:17:33.391255 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:33.391191    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-mfr42" podStartSLOduration=42.883789751 podStartE2EDuration="46.391173685s" totalImagesPullingTime="3.507383934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:33.388949899 +0000 UTC m=+74.882160885" watchObservedRunningTime="2026-09-09 20:17:33.391173685 +0000 UTC m=+74.884384669"
Sep 09 20:17:33.406151 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:33.406122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-mlwpc"]
Sep 09 20:17:34.381097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:34.381060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z7wr5" event={"ID":"48d0e306-fdd9-4e95-ab7c-5ebdbb06f444","Type":"ContainerStarted","Data":"6374b6d638bfc7c4664aeee622ede998c8e068998137d434836df403ba0b1b67"}
Sep 09 20:17:34.381563 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:34.381174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:17:34.382390 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:34.382366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc" event={"ID":"e937672f-c395-458f-890d-aea135850301","Type":"ContainerStarted","Data":"4d2852a1b6d03953584c670a1f91b265e5829ab191e0c1935d42905d66c2eb29"}
Sep 09 20:17:34.383992 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:34.383973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv" event={"ID":"bad2e819-d7a0-4f3a-bbd0-57b076806090","Type":"ContainerStarted","Data":"f338abd39eecaa4f841e61cdc530c2c2ba7f928756fa038b0a7d6697214458f3"}
Sep 09 20:17:34.401953 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:34.401916    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z7wr5" podStartSLOduration=35.896618737 podStartE2EDuration="39.401903558s" totalImagesPullingTime="3.505284821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:34.400310296 +0000 UTC m=+75.893521278" watchObservedRunningTime="2026-09-09 20:17:34.401903558 +0000 UTC m=+75.895114542"
Sep 09 20:17:34.421719 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:34.421681    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-cwptv" podStartSLOduration=43.980375407 podStartE2EDuration="47.421668359s" totalImagesPullingTime="3.441292952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:34.420740685 +0000 UTC m=+75.913951671" watchObservedRunningTime="2026-09-09 20:17:34.421668359 +0000 UTC m=+75.914879343"
Sep 09 20:17:35.389627 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:35.389530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc" event={"ID":"e937672f-c395-458f-890d-aea135850301","Type":"ContainerStarted","Data":"75e334ea81e7c4b3d92b26215562e1e8cb3dc2afbe2f68abb8523f8074b100d7"}
Sep 09 20:17:35.432148 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:35.432101    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-mlwpc" podStartSLOduration=35.797495144 podStartE2EDuration="37.432084575s" totalImagesPullingTime="1.634589431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:35.431719661 +0000 UTC m=+76.924930646" watchObservedRunningTime="2026-09-09 20:17:35.432084575 +0000 UTC m=+76.925295559"
Sep 09 20:17:37.977907 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:37.977866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:37.980264 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:37.980245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aaad6fa0-2376-4fc7-976a-395d763899dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sgqkp\" (UID: \"aaad6fa0-2376-4fc7-976a-395d763899dc\") " pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:38.141556 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:38.141527    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s82pj\""
Sep 09 20:17:38.149644 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:38.149620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sgqkp"
Sep 09 20:17:38.423595 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:38.423573    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sgqkp"]
Sep 09 20:17:38.425999 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:17:38.425972    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaad6fa0_2376_4fc7_976a_395d763899dc.slice/crio-6450cc8609c077e8d373f68e93ab22eea4737ab64ac0fc86a7357d9c70c4a65c WatchSource:0}: Error finding container 6450cc8609c077e8d373f68e93ab22eea4737ab64ac0fc86a7357d9c70c4a65c: Status 404 returned error can't find the container with id 6450cc8609c077e8d373f68e93ab22eea4737ab64ac0fc86a7357d9c70c4a65c
Sep 09 20:17:39.409883 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:39.409800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sgqkp" event={"ID":"aaad6fa0-2376-4fc7-976a-395d763899dc","Type":"ContainerStarted","Data":"c6a328ceea77434a017ee4d17163e7ebf9632ca94d299ab4f5df7ac0c2bb4719"}
Sep 09 20:17:39.409883 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:39.409856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sgqkp" event={"ID":"aaad6fa0-2376-4fc7-976a-395d763899dc","Type":"ContainerStarted","Data":"3a94f5ae84991f9865db75836296cf1d2a95f36bd51e61ffc2bd5d886eb4e947"}
Sep 09 20:17:39.409883 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:39.409874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sgqkp" event={"ID":"aaad6fa0-2376-4fc7-976a-395d763899dc","Type":"ContainerStarted","Data":"6450cc8609c077e8d373f68e93ab22eea4737ab64ac0fc86a7357d9c70c4a65c"}
Sep 09 20:17:40.996712 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:40.996686    2581 scope.go:117] "RemoveContainer" containerID="24a7bb8b971a1f8daa8297d9832dc4da7f92cf5f821dc410ba2ba85a1bca7b13"
Sep 09 20:17:41.420131 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:41.420036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sgqkp" event={"ID":"aaad6fa0-2376-4fc7-976a-395d763899dc","Type":"ContainerStarted","Data":"c64f13d3afeab9c5ae17ab9b788dabafec9a57d693e5fb5f3281e0daaabbe521"}
Sep 09 20:17:41.421925 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:41.421906    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:17:41.422043 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:41.421954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-7c664" event={"ID":"e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68","Type":"ContainerStarted","Data":"db73a81cbe3694567dfa0daa1fb12f3a0d222ff5fd9c857173797696731ba796"}
Sep 09 20:17:41.422201 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:41.422183    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:17:41.437897 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:41.437869    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-7c664"
Sep 09 20:17:41.439664 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:41.439627    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sgqkp" podStartSLOduration=34.478098451 podStartE2EDuration="36.439612241s" totalImagesPullingTime="1.96151379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:17:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:41.439092666 +0000 UTC m=+82.932303651" watchObservedRunningTime="2026-09-09 20:17:41.439612241 +0000 UTC m=+82.932823226"
Sep 09 20:17:42.195977 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.195944    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v2h7c"]
Sep 09 20:17:42.198535 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.198519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.202005 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.201980    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:17:42.202150 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.202012    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:17:42.202150 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.201984    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:17:42.202150 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.202020    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lpcq2\""
Sep 09 20:17:42.205700 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.205679    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:17:42.313117 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.313092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-textfile\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.313249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.313127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a73d7050-de36-4448-8292-9655e23eff84-metrics-client-ca\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.313249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.313147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-tls\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.313249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.313166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a73d7050-de36-4448-8292-9655e23eff84-root\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.313249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.313203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.313249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.313235    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.313487 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.313302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwtkz\" (UniqueName: \"kubernetes.io/projected/a73d7050-de36-4448-8292-9655e23eff84-kube-api-access-qwtkz\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.313487 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.313353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-wtmp\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.313487 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.313378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a73d7050-de36-4448-8292-9655e23eff84-sys\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a73d7050-de36-4448-8292-9655e23eff84-sys\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-textfile\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a73d7050-de36-4448-8292-9655e23eff84-metrics-client-ca\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-tls\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a73d7050-de36-4448-8292-9655e23eff84-root\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a73d7050-de36-4448-8292-9655e23eff84-sys\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a73d7050-de36-4448-8292-9655e23eff84-root\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwtkz\" (UniqueName: \"kubernetes.io/projected/a73d7050-de36-4448-8292-9655e23eff84-kube-api-access-qwtkz\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-wtmp\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414827 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-textfile\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414827 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-wtmp\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.414827 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a73d7050-de36-4448-8292-9655e23eff84-metrics-client-ca\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.415012 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.414991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-accelerators-collector-config\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.416696 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.416673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-tls\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.416791 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.416719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a73d7050-de36-4448-8292-9655e23eff84-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.424579 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.424553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwtkz\" (UniqueName: \"kubernetes.io/projected/a73d7050-de36-4448-8292-9655e23eff84-kube-api-access-qwtkz\") pod \"node-exporter-v2h7c\" (UID: \"a73d7050-de36-4448-8292-9655e23eff84\") " pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.508495 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:42.508422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v2h7c"
Sep 09 20:17:42.519147 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:17:42.519121    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73d7050_de36_4448_8292_9655e23eff84.slice/crio-a2998f4b740696ea755bcb72e91de4e1b9da8d1853e88b5f14a0c2907116fb74 WatchSource:0}: Error finding container a2998f4b740696ea755bcb72e91de4e1b9da8d1853e88b5f14a0c2907116fb74: Status 404 returned error can't find the container with id a2998f4b740696ea755bcb72e91de4e1b9da8d1853e88b5f14a0c2907116fb74
Sep 09 20:17:43.432901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:43.432867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2h7c" event={"ID":"a73d7050-de36-4448-8292-9655e23eff84","Type":"ContainerStarted","Data":"a2998f4b740696ea755bcb72e91de4e1b9da8d1853e88b5f14a0c2907116fb74"}
Sep 09 20:17:44.391661 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:44.391629    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z7wr5"
Sep 09 20:17:44.437968 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:44.437936    2581 generic.go:356] "Generic (PLEG): container finished" podID="a73d7050-de36-4448-8292-9655e23eff84" containerID="c07a472812c3af209d738d5e5e9a919da4a9ba3b789d680be9edda766f2bf5aa" exitCode=0
Sep 09 20:17:44.438415 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:44.438018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2h7c" event={"ID":"a73d7050-de36-4448-8292-9655e23eff84","Type":"ContainerDied","Data":"c07a472812c3af209d738d5e5e9a919da4a9ba3b789d680be9edda766f2bf5aa"}
Sep 09 20:17:45.443721 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:45.443683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2h7c" event={"ID":"a73d7050-de36-4448-8292-9655e23eff84","Type":"ContainerStarted","Data":"d98227e0564d547f49909c9bacd43aecb4562ce53b68a60e8770529bb7715525"}
Sep 09 20:17:45.444128 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:45.443736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v2h7c" event={"ID":"a73d7050-de36-4448-8292-9655e23eff84","Type":"ContainerStarted","Data":"f0c7ed609e55cec41da2ec08d27d0bb72f9f59852077f9e05f0e308372438be0"}
Sep 09 20:17:45.470583 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:45.470533    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v2h7c" podStartSLOduration=2.301768651 podStartE2EDuration="3.470519059s" totalImagesPullingTime="1.099346395s" totalInitContainerRuntime="69.404013ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:45.469261354 +0000 UTC m=+86.962472329" watchObservedRunningTime="2026-09-09 20:17:45.470519059 +0000 UTC m=+86.963730043"
Sep 09 20:17:51.346145 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:51.346114    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:17:53.783124 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.783088    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f9984f7d-8f8lz"]
Sep 09 20:17:53.906077 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.906041    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d664f9fd6-bb7jg"]
Sep 09 20:17:53.910070 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.910048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:53.912733 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.912709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:17:53.912860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.912818    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:17:53.912860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.912712    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:17:53.912860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.912849    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xq6jm\""
Sep 09 20:17:53.913038 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.912914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:17:53.913038 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.912962    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:17:53.913038 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.912968    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:17:53.913196 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.913125    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:17:53.920708 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:53.920689    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d664f9fd6-bb7jg"]
Sep 09 20:17:54.001019 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.000990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmxq6\" (UniqueName: \"kubernetes.io/projected/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-kube-api-access-dmxq6\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.001019 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.001020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-service-ca\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.001212 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.001040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-serving-cert\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.001212 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.001131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-config\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.001212 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.001159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-oauth-config\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.001212 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.001175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-oauth-serving-cert\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.102399 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.102309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-service-ca\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.102399 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.102363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-serving-cert\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.102579 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.102408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-config\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.102579 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.102425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-oauth-config\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.102579 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.102439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-oauth-serving-cert\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.102579 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.102474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmxq6\" (UniqueName: \"kubernetes.io/projected/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-kube-api-access-dmxq6\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.103098 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.103074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-oauth-serving-cert\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.103098 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.103088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-config\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.103098 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.103094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-service-ca\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.105004 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.104979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-oauth-config\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.105186 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.105164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-serving-cert\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.111954 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.111934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmxq6\" (UniqueName: \"kubernetes.io/projected/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-kube-api-access-dmxq6\") pod \"console-5d664f9fd6-bb7jg\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") " pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.219815 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.219784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:17:54.492200 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:54.492177    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d664f9fd6-bb7jg"]
Sep 09 20:17:54.494784 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:17:54.494758    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ff6ca1_7eaf_4379_92c0_6cfcb8ebe37f.slice/crio-1eebfdad084f3c56e50433c2948fa6fdc99b8e069ad586a486540f7bc226dbbf WatchSource:0}: Error finding container 1eebfdad084f3c56e50433c2948fa6fdc99b8e069ad586a486540f7bc226dbbf: Status 404 returned error can't find the container with id 1eebfdad084f3c56e50433c2948fa6fdc99b8e069ad586a486540f7bc226dbbf
Sep 09 20:17:55.488262 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:55.488224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d664f9fd6-bb7jg" event={"ID":"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f","Type":"ContainerStarted","Data":"1eebfdad084f3c56e50433c2948fa6fdc99b8e069ad586a486540f7bc226dbbf"}
Sep 09 20:17:56.310167 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:56.310136    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lf5rw"
Sep 09 20:17:58.502091 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:58.502048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d664f9fd6-bb7jg" event={"ID":"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f","Type":"ContainerStarted","Data":"34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff"}
Sep 09 20:17:58.522670 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:17:58.522602    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d664f9fd6-bb7jg" podStartSLOduration=2.524053553 podStartE2EDuration="5.522582291s" totalImagesPullingTime="2.998528738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:17:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:17:58.522300592 +0000 UTC m=+100.015511582" watchObservedRunningTime="2026-09-09 20:17:58.522582291 +0000 UTC m=+100.015793273"
Sep 09 20:18:04.221000 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.220961    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:18:04.221426 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.221024    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:18:04.225935 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.225911    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:18:04.524378 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.524290    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fc6554db5-4zbcf"]
Sep 09 20:18:04.559845 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.559808    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc6554db5-4zbcf"]
Sep 09 20:18:04.560012 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.559891    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:18:04.560012 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.559933    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.567440 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.567408    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:18:04.574118 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.574097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-console-config\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.574233 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.574124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-serving-cert\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.574422 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.574401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-trusted-ca-bundle\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.574472 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.574439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-oauth-serving-cert\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.574585 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.574476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-oauth-config\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.574585 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.574491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-service-ca\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.574585 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.574508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf9dj\" (UniqueName: \"kubernetes.io/projected/453c0c85-3d94-451e-8629-d483c3d2cae3-kube-api-access-gf9dj\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.675574 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.675536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-serving-cert\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.675725 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.675591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-trusted-ca-bundle\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.675725 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.675660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-oauth-serving-cert\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.675799 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.675732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-oauth-config\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.675799 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.675749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-service-ca\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.675799 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.675765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf9dj\" (UniqueName: \"kubernetes.io/projected/453c0c85-3d94-451e-8629-d483c3d2cae3-kube-api-access-gf9dj\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.675966 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.675814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-console-config\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.676587 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.676516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-service-ca\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.676587 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.676565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-console-config\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.676587 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.676574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-oauth-serving-cert\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.676815 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.676767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-trusted-ca-bundle\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.678203 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.678180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-oauth-config\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.678322 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.678305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-serving-cert\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.683829 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.683809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf9dj\" (UniqueName: \"kubernetes.io/projected/453c0c85-3d94-451e-8629-d483c3d2cae3-kube-api-access-gf9dj\") pod \"console-fc6554db5-4zbcf\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") " pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:04.869764 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:04.869672    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:05.172679 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:05.172656    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fc6554db5-4zbcf"]
Sep 09 20:18:05.175116 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:18:05.175090    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453c0c85_3d94_451e_8629_d483c3d2cae3.slice/crio-633de79f817ecbd9572da5f03692352fa705e3a5d9f9f372cd600c3ddf41745b WatchSource:0}: Error finding container 633de79f817ecbd9572da5f03692352fa705e3a5d9f9f372cd600c3ddf41745b: Status 404 returned error can't find the container with id 633de79f817ecbd9572da5f03692352fa705e3a5d9f9f372cd600c3ddf41745b
Sep 09 20:18:05.529839 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:05.529803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc6554db5-4zbcf" event={"ID":"453c0c85-3d94-451e-8629-d483c3d2cae3","Type":"ContainerStarted","Data":"031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7"}
Sep 09 20:18:05.530324 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:05.529858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc6554db5-4zbcf" event={"ID":"453c0c85-3d94-451e-8629-d483c3d2cae3","Type":"ContainerStarted","Data":"633de79f817ecbd9572da5f03692352fa705e3a5d9f9f372cd600c3ddf41745b"}
Sep 09 20:18:05.551853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:05.551800    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fc6554db5-4zbcf" podStartSLOduration=1.551784108 podStartE2EDuration="1.551784108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:05.550524589 +0000 UTC m=+107.043735573" watchObservedRunningTime="2026-09-09 20:18:05.551784108 +0000 UTC m=+107.044995092"
Sep 09 20:18:14.870823 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:14.870783    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:14.871321 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:14.870843    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:14.875769 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:14.875742    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:15.570176 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:15.570148    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:18:15.616750 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:15.616715    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d664f9fd6-bb7jg"]
Sep 09 20:18:18.803359 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:18.803307    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz" podUID="306d365e-c1ae-4944-868f-844b11fd59f4" containerName="registry" containerID="cri-o://d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571" gracePeriod=30
Sep 09 20:18:19.048700 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.048676    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:18:19.088783 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.088703    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-installation-pull-secrets\") pod \"306d365e-c1ae-4944-868f-844b11fd59f4\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") "
Sep 09 20:18:19.088783 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.088761    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-bound-sa-token\") pod \"306d365e-c1ae-4944-868f-844b11fd59f4\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") "
Sep 09 20:18:19.088979 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.088863    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7vkj\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-kube-api-access-f7vkj\") pod \"306d365e-c1ae-4944-868f-844b11fd59f4\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") "
Sep 09 20:18:19.088979 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.088906    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306d365e-c1ae-4944-868f-844b11fd59f4-ca-trust-extracted\") pod \"306d365e-c1ae-4944-868f-844b11fd59f4\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") "
Sep 09 20:18:19.088979 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.088940    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls\") pod \"306d365e-c1ae-4944-868f-844b11fd59f4\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") "
Sep 09 20:18:19.089142 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.089000    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-registry-certificates\") pod \"306d365e-c1ae-4944-868f-844b11fd59f4\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") "
Sep 09 20:18:19.089142 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.089032    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-trusted-ca\") pod \"306d365e-c1ae-4944-868f-844b11fd59f4\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") "
Sep 09 20:18:19.089142 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.089064    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-image-registry-private-configuration\") pod \"306d365e-c1ae-4944-868f-844b11fd59f4\" (UID: \"306d365e-c1ae-4944-868f-844b11fd59f4\") "
Sep 09 20:18:19.090511 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.089574    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "306d365e-c1ae-4944-868f-844b11fd59f4" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:19.090511 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.089879    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "306d365e-c1ae-4944-868f-844b11fd59f4" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:19.092357 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.092280    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "306d365e-c1ae-4944-868f-844b11fd59f4" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:19.092629 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.092603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-kube-api-access-f7vkj" (OuterVolumeSpecName: "kube-api-access-f7vkj") pod "306d365e-c1ae-4944-868f-844b11fd59f4" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4"). InnerVolumeSpecName "kube-api-access-f7vkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:19.092947 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.092924    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "306d365e-c1ae-4944-868f-844b11fd59f4" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:19.093023 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.092998    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "306d365e-c1ae-4944-868f-844b11fd59f4" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:19.093260 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.093239    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "306d365e-c1ae-4944-868f-844b11fd59f4" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:19.101468 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.101434    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306d365e-c1ae-4944-868f-844b11fd59f4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "306d365e-c1ae-4944-868f-844b11fd59f4" (UID: "306d365e-c1ae-4944-868f-844b11fd59f4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:18:19.190086 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.190054    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-trusted-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:19.190086 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.190084    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-image-registry-private-configuration\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:19.190086 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.190096    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/306d365e-c1ae-4944-868f-844b11fd59f4-installation-pull-secrets\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:19.190306 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.190106    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-bound-sa-token\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:19.190306 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.190115    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7vkj\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-kube-api-access-f7vkj\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:19.190306 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.190122    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/306d365e-c1ae-4944-868f-844b11fd59f4-ca-trust-extracted\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:19.190306 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.190131    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/306d365e-c1ae-4944-868f-844b11fd59f4-registry-tls\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:19.190306 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.190139    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/306d365e-c1ae-4944-868f-844b11fd59f4-registry-certificates\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:19.581252 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.581213    2581 generic.go:356] "Generic (PLEG): container finished" podID="306d365e-c1ae-4944-868f-844b11fd59f4" containerID="d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571" exitCode=0
Sep 09 20:18:19.581448 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.581276    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz"
Sep 09 20:18:19.581448 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.581299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz" event={"ID":"306d365e-c1ae-4944-868f-844b11fd59f4","Type":"ContainerDied","Data":"d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571"}
Sep 09 20:18:19.581448 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.581419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84f9984f7d-8f8lz" event={"ID":"306d365e-c1ae-4944-868f-844b11fd59f4","Type":"ContainerDied","Data":"1241d4afeb674e77de7b405415f511340a54b6284329f0010375c4d8f19c8f4b"}
Sep 09 20:18:19.581448 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.581445    2581 scope.go:117] "RemoveContainer" containerID="d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571"
Sep 09 20:18:19.590198 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.590176    2581 scope.go:117] "RemoveContainer" containerID="d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571"
Sep 09 20:18:19.590507 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:18:19.590484    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571\": container with ID starting with d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571 not found: ID does not exist" containerID="d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571"
Sep 09 20:18:19.590554 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.590515    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571"} err="failed to get container status \"d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571\": rpc error: code = NotFound desc = could not find container \"d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571\": container with ID starting with d9a9a5627578e85479bd06a838460e86e16871562a7d7e34dc9491b9b2978571 not found: ID does not exist"
Sep 09 20:18:19.605261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.605240    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84f9984f7d-8f8lz"]
Sep 09 20:18:19.609931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:19.609913    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84f9984f7d-8f8lz"]
Sep 09 20:18:20.586571 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:20.586479    2581 generic.go:356] "Generic (PLEG): container finished" podID="670f84b1-5f23-44cf-9b05-e4a3e603b7fa" containerID="6527fd4f50f61f4c985486e9a12252fa05e5b0c43e6c872ba64283751411819d" exitCode=0
Sep 09 20:18:20.587012 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:20.586561    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt" event={"ID":"670f84b1-5f23-44cf-9b05-e4a3e603b7fa","Type":"ContainerDied","Data":"6527fd4f50f61f4c985486e9a12252fa05e5b0c43e6c872ba64283751411819d"}
Sep 09 20:18:20.587012 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:20.586991    2581 scope.go:117] "RemoveContainer" containerID="6527fd4f50f61f4c985486e9a12252fa05e5b0c43e6c872ba64283751411819d"
Sep 09 20:18:20.589364 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:20.589325    2581 generic.go:356] "Generic (PLEG): container finished" podID="a32e9cee-8743-486e-a042-41ae301e7f11" containerID="b6c358e8d21302d62c2f5d22bbf29dcab22f446ebba0c832ba0e25e9abc73f3a" exitCode=0
Sep 09 20:18:20.589447 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:20.589376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b" event={"ID":"a32e9cee-8743-486e-a042-41ae301e7f11","Type":"ContainerDied","Data":"b6c358e8d21302d62c2f5d22bbf29dcab22f446ebba0c832ba0e25e9abc73f3a"}
Sep 09 20:18:20.589650 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:20.589637    2581 scope.go:117] "RemoveContainer" containerID="b6c358e8d21302d62c2f5d22bbf29dcab22f446ebba0c832ba0e25e9abc73f3a"
Sep 09 20:18:21.003510 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:21.003469    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="306d365e-c1ae-4944-868f-844b11fd59f4" path="/var/lib/kubelet/pods/306d365e-c1ae-4944-868f-844b11fd59f4/volumes"
Sep 09 20:18:21.595622 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:21.595588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-gx45b" event={"ID":"a32e9cee-8743-486e-a042-41ae301e7f11","Type":"ContainerStarted","Data":"f0e00e261a5e6457a6ab429725a396330387a2a0f6755dd05379f576af9ed507"}
Sep 09 20:18:21.597467 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:21.597444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-tbhnt" event={"ID":"670f84b1-5f23-44cf-9b05-e4a3e603b7fa","Type":"ContainerStarted","Data":"c9a1a6dc7a65a6621fb6f230b01cd38eae6e687eb4ba825dee430b7ac1766fbd"}
Sep 09 20:18:30.631644 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:30.631614    2581 generic.go:356] "Generic (PLEG): container finished" podID="83a81b0e-8d00-470d-b1b3-28b2832b2746" containerID="d7b5b5d6e2a0cd0c1c487ae6820820e396b17b037fa7fd8c9baad8ddb021f282" exitCode=0
Sep 09 20:18:30.632073 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:30.631693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9" event={"ID":"83a81b0e-8d00-470d-b1b3-28b2832b2746","Type":"ContainerDied","Data":"d7b5b5d6e2a0cd0c1c487ae6820820e396b17b037fa7fd8c9baad8ddb021f282"}
Sep 09 20:18:30.632073 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:30.632016    2581 scope.go:117] "RemoveContainer" containerID="d7b5b5d6e2a0cd0c1c487ae6820820e396b17b037fa7fd8c9baad8ddb021f282"
Sep 09 20:18:31.638067 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:31.638031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-psxc9" event={"ID":"83a81b0e-8d00-470d-b1b3-28b2832b2746","Type":"ContainerStarted","Data":"bb0c103d590faf5ba0e9d631b20d3c8c4eea8d7cdd4bdbd28e45c1c3162d2966"}
Sep 09 20:18:40.643422 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.643333    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d664f9fd6-bb7jg" podUID="57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" containerName="console" containerID="cri-o://34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff" gracePeriod=15
Sep 09 20:18:40.883744 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.883721    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:18:40.968715 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.968686    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-oauth-config\") pod \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") "
Sep 09 20:18:40.968871 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.968730    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-oauth-serving-cert\") pod \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") "
Sep 09 20:18:40.968871 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.968757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-config\") pod \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") "
Sep 09 20:18:40.968871 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.968812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-serving-cert\") pod \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") "
Sep 09 20:18:40.968871 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.968831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-service-ca\") pod \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") "
Sep 09 20:18:40.968871 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.968856    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmxq6\" (UniqueName: \"kubernetes.io/projected/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-kube-api-access-dmxq6\") pod \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\" (UID: \"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f\") "
Sep 09 20:18:40.969225 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.969182    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" (UID: "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:40.969296 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.969228    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-config" (OuterVolumeSpecName: "console-config") pod "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" (UID: "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:40.969296 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.969234    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-service-ca" (OuterVolumeSpecName: "service-ca") pod "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" (UID: "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:18:40.971168 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.971139    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" (UID: "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:40.971168 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.971136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-kube-api-access-dmxq6" (OuterVolumeSpecName: "kube-api-access-dmxq6") pod "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" (UID: "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f"). InnerVolumeSpecName "kube-api-access-dmxq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:40.971314 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:40.971225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" (UID: "57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:41.069676 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.069651    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-config\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:41.069826 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.069759    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-serving-cert\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:41.069901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.069855    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:41.069901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.069870    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmxq6\" (UniqueName: \"kubernetes.io/projected/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-kube-api-access-dmxq6\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:41.069901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.069880    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-console-oauth-config\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:41.069901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.069888    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f-oauth-serving-cert\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:18:41.680412 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.680375    2581 generic.go:356] "Generic (PLEG): container finished" podID="57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" containerID="34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff" exitCode=0
Sep 09 20:18:41.680954 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.680414    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d664f9fd6-bb7jg"
Sep 09 20:18:41.680954 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.680447    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d664f9fd6-bb7jg" event={"ID":"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f","Type":"ContainerDied","Data":"34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff"}
Sep 09 20:18:41.680954 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.680510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d664f9fd6-bb7jg" event={"ID":"57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f","Type":"ContainerDied","Data":"1eebfdad084f3c56e50433c2948fa6fdc99b8e069ad586a486540f7bc226dbbf"}
Sep 09 20:18:41.680954 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.680539    2581 scope.go:117] "RemoveContainer" containerID="34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff"
Sep 09 20:18:41.689151 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.689134    2581 scope.go:117] "RemoveContainer" containerID="34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff"
Sep 09 20:18:41.689435 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:18:41.689419    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff\": container with ID starting with 34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff not found: ID does not exist" containerID="34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff"
Sep 09 20:18:41.689485 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.689444    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff"} err="failed to get container status \"34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff\": rpc error: code = NotFound desc = could not find container \"34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff\": container with ID starting with 34238bb14f6e7382e3a37bc2dc876e4762e21cff06ddcb63a7248d08ff8662ff not found: ID does not exist"
Sep 09 20:18:41.701753 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.701728    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d664f9fd6-bb7jg"]
Sep 09 20:18:41.706524 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:41.706503    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d664f9fd6-bb7jg"]
Sep 09 20:18:43.001654 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:18:43.001623    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" path="/var/lib/kubelet/pods/57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f/volumes"
Sep 09 20:19:10.284961 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.284881    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bfc6b877f-r4pv4"]
Sep 09 20:19:10.285432 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.285297    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="306d365e-c1ae-4944-868f-844b11fd59f4" containerName="registry"
Sep 09 20:19:10.285432 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.285314    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="306d365e-c1ae-4944-868f-844b11fd59f4" containerName="registry"
Sep 09 20:19:10.285432 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.285326    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" containerName="console"
Sep 09 20:19:10.285432 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.285336    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" containerName="console"
Sep 09 20:19:10.285830 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.285453    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="306d365e-c1ae-4944-868f-844b11fd59f4" containerName="registry"
Sep 09 20:19:10.285830 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.285466    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ff6ca1-7eaf-4379-92c0-6cfcb8ebe37f" containerName="console"
Sep 09 20:19:10.288285 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.288263    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.294108 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.294078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-trusted-ca-bundle\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.294253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.294120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-service-ca\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.294253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.294177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-console-config\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.294253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.294199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-serving-cert\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.294474 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.294274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-oauth-serving-cert\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.294474 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.294308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9568\" (UniqueName: \"kubernetes.io/projected/d8808357-c419-4c15-9da2-1d45fd70e882-kube-api-access-n9568\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.294474 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.294375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-oauth-config\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.304399 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.304365    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bfc6b877f-r4pv4"]
Sep 09 20:19:10.394865 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.394830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-oauth-serving-cert\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.394865 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.394869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9568\" (UniqueName: \"kubernetes.io/projected/d8808357-c419-4c15-9da2-1d45fd70e882-kube-api-access-n9568\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.395120 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.394895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-oauth-config\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.395120 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.395013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-trusted-ca-bundle\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.395120 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.395057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-service-ca\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.395120 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.395108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-console-config\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.395316 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.395132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-serving-cert\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.395704 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.395674    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-oauth-serving-cert\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.395821 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.395760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-trusted-ca-bundle\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.395895 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.395857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-console-config\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.395895 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.395857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-service-ca\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.397463 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.397441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-oauth-config\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.397582 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.397566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-serving-cert\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.403466 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.403444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9568\" (UniqueName: \"kubernetes.io/projected/d8808357-c419-4c15-9da2-1d45fd70e882-kube-api-access-n9568\") pod \"console-bfc6b877f-r4pv4\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") " pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.599410 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.599298    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:10.870570 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:10.870495    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bfc6b877f-r4pv4"]
Sep 09 20:19:10.873395 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:19:10.873364    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8808357_c419_4c15_9da2_1d45fd70e882.slice/crio-ec0eb821ed4b1a68116529d7d16d4e3020c539f05d09634cd2d0ff38b27ac23a WatchSource:0}: Error finding container ec0eb821ed4b1a68116529d7d16d4e3020c539f05d09634cd2d0ff38b27ac23a: Status 404 returned error can't find the container with id ec0eb821ed4b1a68116529d7d16d4e3020c539f05d09634cd2d0ff38b27ac23a
Sep 09 20:19:11.796361 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:11.796309    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bfc6b877f-r4pv4" event={"ID":"d8808357-c419-4c15-9da2-1d45fd70e882","Type":"ContainerStarted","Data":"c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32"}
Sep 09 20:19:11.796759 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:11.796385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bfc6b877f-r4pv4" event={"ID":"d8808357-c419-4c15-9da2-1d45fd70e882","Type":"ContainerStarted","Data":"ec0eb821ed4b1a68116529d7d16d4e3020c539f05d09634cd2d0ff38b27ac23a"}
Sep 09 20:19:11.823446 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:11.823401    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bfc6b877f-r4pv4" podStartSLOduration=1.82338613 podStartE2EDuration="1.82338613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:11.823012769 +0000 UTC m=+173.316223752" watchObservedRunningTime="2026-09-09 20:19:11.82338613 +0000 UTC m=+173.316597111"
Sep 09 20:19:20.599440 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:20.599406    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:20.599848 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:20.599504    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:20.604602 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:20.604578    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:20.835410 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:20.835381    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:19:20.887443 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:20.887367    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fc6554db5-4zbcf"]
Sep 09 20:19:45.915190 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:45.915127    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fc6554db5-4zbcf" podUID="453c0c85-3d94-451e-8629-d483c3d2cae3" containerName="console" containerID="cri-o://031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7" gracePeriod=15
Sep 09 20:19:46.162671 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.162647    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:19:46.260784 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.260755    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-oauth-config\") pod \"453c0c85-3d94-451e-8629-d483c3d2cae3\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") "
Sep 09 20:19:46.260974 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.260811    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-serving-cert\") pod \"453c0c85-3d94-451e-8629-d483c3d2cae3\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") "
Sep 09 20:19:46.260974 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.260838    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-console-config\") pod \"453c0c85-3d94-451e-8629-d483c3d2cae3\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") "
Sep 09 20:19:46.260974 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.260857    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-oauth-serving-cert\") pod \"453c0c85-3d94-451e-8629-d483c3d2cae3\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") "
Sep 09 20:19:46.261146 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.260988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf9dj\" (UniqueName: \"kubernetes.io/projected/453c0c85-3d94-451e-8629-d483c3d2cae3-kube-api-access-gf9dj\") pod \"453c0c85-3d94-451e-8629-d483c3d2cae3\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") "
Sep 09 20:19:46.261146 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.261029    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-service-ca\") pod \"453c0c85-3d94-451e-8629-d483c3d2cae3\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") "
Sep 09 20:19:46.261146 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.261068    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-trusted-ca-bundle\") pod \"453c0c85-3d94-451e-8629-d483c3d2cae3\" (UID: \"453c0c85-3d94-451e-8629-d483c3d2cae3\") "
Sep 09 20:19:46.261419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.261267    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "453c0c85-3d94-451e-8629-d483c3d2cae3" (UID: "453c0c85-3d94-451e-8629-d483c3d2cae3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:19:46.261419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.261386    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-console-config" (OuterVolumeSpecName: "console-config") pod "453c0c85-3d94-451e-8629-d483c3d2cae3" (UID: "453c0c85-3d94-451e-8629-d483c3d2cae3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:19:46.261606 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.261527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-service-ca" (OuterVolumeSpecName: "service-ca") pod "453c0c85-3d94-451e-8629-d483c3d2cae3" (UID: "453c0c85-3d94-451e-8629-d483c3d2cae3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:19:46.261693 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.261673    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "453c0c85-3d94-451e-8629-d483c3d2cae3" (UID: "453c0c85-3d94-451e-8629-d483c3d2cae3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:19:46.263048 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.263026    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "453c0c85-3d94-451e-8629-d483c3d2cae3" (UID: "453c0c85-3d94-451e-8629-d483c3d2cae3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:19:46.263679 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.263657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "453c0c85-3d94-451e-8629-d483c3d2cae3" (UID: "453c0c85-3d94-451e-8629-d483c3d2cae3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:19:46.263679 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.263668    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453c0c85-3d94-451e-8629-d483c3d2cae3-kube-api-access-gf9dj" (OuterVolumeSpecName: "kube-api-access-gf9dj") pod "453c0c85-3d94-451e-8629-d483c3d2cae3" (UID: "453c0c85-3d94-451e-8629-d483c3d2cae3"). InnerVolumeSpecName "kube-api-access-gf9dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:46.362145 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.362106    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-serving-cert\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:19:46.362145 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.362138    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-console-config\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:19:46.362145 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.362151    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-oauth-serving-cert\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:19:46.362415 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.362164    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf9dj\" (UniqueName: \"kubernetes.io/projected/453c0c85-3d94-451e-8629-d483c3d2cae3-kube-api-access-gf9dj\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:19:46.362415 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.362177    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:19:46.362415 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.362188    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/453c0c85-3d94-451e-8629-d483c3d2cae3-trusted-ca-bundle\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:19:46.362415 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.362199    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/453c0c85-3d94-451e-8629-d483c3d2cae3-console-oauth-config\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:19:46.928855 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.928813    2581 generic.go:356] "Generic (PLEG): container finished" podID="453c0c85-3d94-451e-8629-d483c3d2cae3" containerID="031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7" exitCode=0
Sep 09 20:19:46.929246 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.928891    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fc6554db5-4zbcf"
Sep 09 20:19:46.929246 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.928900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc6554db5-4zbcf" event={"ID":"453c0c85-3d94-451e-8629-d483c3d2cae3","Type":"ContainerDied","Data":"031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7"}
Sep 09 20:19:46.929246 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.928983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fc6554db5-4zbcf" event={"ID":"453c0c85-3d94-451e-8629-d483c3d2cae3","Type":"ContainerDied","Data":"633de79f817ecbd9572da5f03692352fa705e3a5d9f9f372cd600c3ddf41745b"}
Sep 09 20:19:46.929246 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.929003    2581 scope.go:117] "RemoveContainer" containerID="031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7"
Sep 09 20:19:46.937980 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.937961    2581 scope.go:117] "RemoveContainer" containerID="031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7"
Sep 09 20:19:46.938268 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:19:46.938250    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7\": container with ID starting with 031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7 not found: ID does not exist" containerID="031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7"
Sep 09 20:19:46.938330 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.938281    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7"} err="failed to get container status \"031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7\": rpc error: code = NotFound desc = could not find container \"031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7\": container with ID starting with 031decf6573b4fa66c832fc786d4a6fd9f1929bbcfb56a339d6053bbb40528a7 not found: ID does not exist"
Sep 09 20:19:46.950186 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.950165    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fc6554db5-4zbcf"]
Sep 09 20:19:46.954624 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:46.954603    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fc6554db5-4zbcf"]
Sep 09 20:19:47.000668 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:19:47.000643    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="453c0c85-3d94-451e-8629-d483c3d2cae3" path="/var/lib/kubelet/pods/453c0c85-3d94-451e-8629-d483c3d2cae3/volumes"
Sep 09 20:20:21.897329 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:21.897297    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8567f7f74-bhxx7"]
Sep 09 20:20:21.897860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:21.897615    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453c0c85-3d94-451e-8629-d483c3d2cae3" containerName="console"
Sep 09 20:20:21.897860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:21.897630    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="453c0c85-3d94-451e-8629-d483c3d2cae3" containerName="console"
Sep 09 20:20:21.897860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:21.897695    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="453c0c85-3d94-451e-8629-d483c3d2cae3" containerName="console"
Sep 09 20:20:21.901549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:21.901528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:21.911184 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:21.911162    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8567f7f74-bhxx7"]
Sep 09 20:20:22.035362 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.035308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55j4j\" (UniqueName: \"kubernetes.io/projected/c8f6a62f-4e94-47e7-9dd4-eff95079e538-kube-api-access-55j4j\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.035518 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.035448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-oauth-config\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.035518 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.035508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-trusted-ca-bundle\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.035593 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.035539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-config\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.035593 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.035572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-oauth-serving-cert\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.035655 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.035611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-serving-cert\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.035655 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.035636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-service-ca\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.136426 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.136390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-oauth-config\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.136612 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.136447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-trusted-ca-bundle\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.136679 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.136614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-config\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.136679 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.136646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-oauth-serving-cert\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.136679 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.136668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-serving-cert\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.136830 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.136690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-service-ca\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.136830 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.136716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j4j\" (UniqueName: \"kubernetes.io/projected/c8f6a62f-4e94-47e7-9dd4-eff95079e538-kube-api-access-55j4j\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.137363 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.137322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-config\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.137478 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.137413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-oauth-serving-cert\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.137478 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.137426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-service-ca\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.137606 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.137589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-trusted-ca-bundle\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.139583 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.139565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-serving-cert\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.139697 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.139678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-oauth-config\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.145071 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.145043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55j4j\" (UniqueName: \"kubernetes.io/projected/c8f6a62f-4e94-47e7-9dd4-eff95079e538-kube-api-access-55j4j\") pod \"console-8567f7f74-bhxx7\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") " pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.211462 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.211434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:22.485227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:22.485156    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8567f7f74-bhxx7"]
Sep 09 20:20:22.488495 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:20:22.488462    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f6a62f_4e94_47e7_9dd4_eff95079e538.slice/crio-dc2f1e9c0dac58929c8b7067e6fcff57b3de03b9005c4e0ffcc3d39a7739512d WatchSource:0}: Error finding container dc2f1e9c0dac58929c8b7067e6fcff57b3de03b9005c4e0ffcc3d39a7739512d: Status 404 returned error can't find the container with id dc2f1e9c0dac58929c8b7067e6fcff57b3de03b9005c4e0ffcc3d39a7739512d
Sep 09 20:20:23.062501 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:23.062466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8567f7f74-bhxx7" event={"ID":"c8f6a62f-4e94-47e7-9dd4-eff95079e538","Type":"ContainerStarted","Data":"f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d"}
Sep 09 20:20:23.062877 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:23.062510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8567f7f74-bhxx7" event={"ID":"c8f6a62f-4e94-47e7-9dd4-eff95079e538","Type":"ContainerStarted","Data":"dc2f1e9c0dac58929c8b7067e6fcff57b3de03b9005c4e0ffcc3d39a7739512d"}
Sep 09 20:20:23.081745 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:23.081694    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8567f7f74-bhxx7" podStartSLOduration=2.081679971 podStartE2EDuration="2.081679971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:20:23.080402044 +0000 UTC m=+244.573613031" watchObservedRunningTime="2026-09-09 20:20:23.081679971 +0000 UTC m=+244.574890954"
Sep 09 20:20:32.213311 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:32.213258    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:32.213893 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:32.213422    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:32.218172 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:32.218148    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:33.104018 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:33.103989    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:20:33.153373 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:33.153324    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bfc6b877f-r4pv4"]
Sep 09 20:20:58.180146 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.180083    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bfc6b877f-r4pv4" podUID="d8808357-c419-4c15-9da2-1d45fd70e882" containerName="console" containerID="cri-o://c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32" gracePeriod=15
Sep 09 20:20:58.411177 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.411151    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:20:58.502128 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.502047    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-console-config\") pod \"d8808357-c419-4c15-9da2-1d45fd70e882\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") "
Sep 09 20:20:58.502128 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.502090    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9568\" (UniqueName: \"kubernetes.io/projected/d8808357-c419-4c15-9da2-1d45fd70e882-kube-api-access-n9568\") pod \"d8808357-c419-4c15-9da2-1d45fd70e882\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") "
Sep 09 20:20:58.502128 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.502119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-oauth-serving-cert\") pod \"d8808357-c419-4c15-9da2-1d45fd70e882\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") "
Sep 09 20:20:58.502529 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.502166    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-trusted-ca-bundle\") pod \"d8808357-c419-4c15-9da2-1d45fd70e882\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") "
Sep 09 20:20:58.502529 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.502193    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-serving-cert\") pod \"d8808357-c419-4c15-9da2-1d45fd70e882\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") "
Sep 09 20:20:58.502529 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.502218    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-service-ca\") pod \"d8808357-c419-4c15-9da2-1d45fd70e882\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") "
Sep 09 20:20:58.502529 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.502244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-oauth-config\") pod \"d8808357-c419-4c15-9da2-1d45fd70e882\" (UID: \"d8808357-c419-4c15-9da2-1d45fd70e882\") "
Sep 09 20:20:58.502709 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.502569    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-console-config" (OuterVolumeSpecName: "console-config") pod "d8808357-c419-4c15-9da2-1d45fd70e882" (UID: "d8808357-c419-4c15-9da2-1d45fd70e882"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:20:58.502709 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.502684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d8808357-c419-4c15-9da2-1d45fd70e882" (UID: "d8808357-c419-4c15-9da2-1d45fd70e882"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:20:58.502785 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.502700    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-service-ca" (OuterVolumeSpecName: "service-ca") pod "d8808357-c419-4c15-9da2-1d45fd70e882" (UID: "d8808357-c419-4c15-9da2-1d45fd70e882"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:20:58.502904 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.502877    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d8808357-c419-4c15-9da2-1d45fd70e882" (UID: "d8808357-c419-4c15-9da2-1d45fd70e882"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:20:58.504386 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.504338    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8808357-c419-4c15-9da2-1d45fd70e882-kube-api-access-n9568" (OuterVolumeSpecName: "kube-api-access-n9568") pod "d8808357-c419-4c15-9da2-1d45fd70e882" (UID: "d8808357-c419-4c15-9da2-1d45fd70e882"). InnerVolumeSpecName "kube-api-access-n9568". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:20:58.504539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.504522    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d8808357-c419-4c15-9da2-1d45fd70e882" (UID: "d8808357-c419-4c15-9da2-1d45fd70e882"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:20:58.504594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.504536    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d8808357-c419-4c15-9da2-1d45fd70e882" (UID: "d8808357-c419-4c15-9da2-1d45fd70e882"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:20:58.603049 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.603006    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9568\" (UniqueName: \"kubernetes.io/projected/d8808357-c419-4c15-9da2-1d45fd70e882-kube-api-access-n9568\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:20:58.603049 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.603046    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-oauth-serving-cert\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:20:58.603049 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.603057    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-trusted-ca-bundle\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:20:58.603049 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.603065    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-serving-cert\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:20:58.603273 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.603073    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:20:58.603273 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.603081    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8808357-c419-4c15-9da2-1d45fd70e882-console-oauth-config\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:20:58.603273 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:58.603088    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8808357-c419-4c15-9da2-1d45fd70e882-console-config\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:20:59.200011 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:59.199971    2581 generic.go:356] "Generic (PLEG): container finished" podID="d8808357-c419-4c15-9da2-1d45fd70e882" containerID="c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32" exitCode=0
Sep 09 20:20:59.200497 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:59.200029    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bfc6b877f-r4pv4"
Sep 09 20:20:59.200497 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:59.200050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bfc6b877f-r4pv4" event={"ID":"d8808357-c419-4c15-9da2-1d45fd70e882","Type":"ContainerDied","Data":"c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32"}
Sep 09 20:20:59.200497 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:59.200106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bfc6b877f-r4pv4" event={"ID":"d8808357-c419-4c15-9da2-1d45fd70e882","Type":"ContainerDied","Data":"ec0eb821ed4b1a68116529d7d16d4e3020c539f05d09634cd2d0ff38b27ac23a"}
Sep 09 20:20:59.200497 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:59.200134    2581 scope.go:117] "RemoveContainer" containerID="c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32"
Sep 09 20:20:59.208779 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:59.208761    2581 scope.go:117] "RemoveContainer" containerID="c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32"
Sep 09 20:20:59.209075 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:20:59.209058    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32\": container with ID starting with c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32 not found: ID does not exist" containerID="c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32"
Sep 09 20:20:59.209131 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:59.209083    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32"} err="failed to get container status \"c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32\": rpc error: code = NotFound desc = could not find container \"c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32\": container with ID starting with c527948c7656d4ae74f0417c3250ee8082235203d6960a45308a67baeda4ca32 not found: ID does not exist"
Sep 09 20:20:59.216596 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:59.216571    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bfc6b877f-r4pv4"]
Sep 09 20:20:59.220226 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:20:59.220207    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bfc6b877f-r4pv4"]
Sep 09 20:21:01.001513 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:01.001479    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8808357-c419-4c15-9da2-1d45fd70e882" path="/var/lib/kubelet/pods/d8808357-c419-4c15-9da2-1d45fd70e882/volumes"
Sep 09 20:21:18.908571 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:18.908541    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:21:18.912997 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:18.912973    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:21:18.918735 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:18.918716    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:21:21.837319 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.837281    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"]
Sep 09 20:21:21.839561 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.837583    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8808357-c419-4c15-9da2-1d45fd70e882" containerName="console"
Sep 09 20:21:21.839561 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.837596    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8808357-c419-4c15-9da2-1d45fd70e882" containerName="console"
Sep 09 20:21:21.839561 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.837655    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8808357-c419-4c15-9da2-1d45fd70e882" containerName="console"
Sep 09 20:21:21.840603 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.840587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:21.843325 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.843294    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mwfck\""
Sep 09 20:21:21.843325 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.843317    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:21:21.844191 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.844173    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:21:21.854173 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.854156    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"]
Sep 09 20:21:21.985677 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.985642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv5rh\" (UniqueName: \"kubernetes.io/projected/029eacaf-951a-4c17-9a23-6995fe9c5fdc-kube-api-access-qv5rh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:21.985677 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.985678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:21.985913 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:21.985720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:22.086539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:22.086505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv5rh\" (UniqueName: \"kubernetes.io/projected/029eacaf-951a-4c17-9a23-6995fe9c5fdc-kube-api-access-qv5rh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:22.086664 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:22.086546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:22.086664 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:22.086600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:22.086967 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:22.086949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:22.087033 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:22.086968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:22.097069 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:22.097012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv5rh\" (UniqueName: \"kubernetes.io/projected/029eacaf-951a-4c17-9a23-6995fe9c5fdc-kube-api-access-qv5rh\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:22.149241 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:22.149209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:22.421124 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:22.421039    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"]
Sep 09 20:21:22.423855 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:21:22.423810    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029eacaf_951a_4c17_9a23_6995fe9c5fdc.slice/crio-6dcab805a876b7ea88fad7e64c6b5ee17003a4199b76a9c09696c3aa2e8e8f02 WatchSource:0}: Error finding container 6dcab805a876b7ea88fad7e64c6b5ee17003a4199b76a9c09696c3aa2e8e8f02: Status 404 returned error can't find the container with id 6dcab805a876b7ea88fad7e64c6b5ee17003a4199b76a9c09696c3aa2e8e8f02
Sep 09 20:21:22.426036 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:22.426017    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:21:23.292838 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:23.292804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484" event={"ID":"029eacaf-951a-4c17-9a23-6995fe9c5fdc","Type":"ContainerStarted","Data":"6dcab805a876b7ea88fad7e64c6b5ee17003a4199b76a9c09696c3aa2e8e8f02"}
Sep 09 20:21:30.322099 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:30.322054    2581 generic.go:356] "Generic (PLEG): container finished" podID="029eacaf-951a-4c17-9a23-6995fe9c5fdc" containerID="7534b0a2eed0d31dbf04c2870a8c423cc27e344975d0bb94d783655b24ea546d" exitCode=0
Sep 09 20:21:30.322511 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:30.322126    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484" event={"ID":"029eacaf-951a-4c17-9a23-6995fe9c5fdc","Type":"ContainerDied","Data":"7534b0a2eed0d31dbf04c2870a8c423cc27e344975d0bb94d783655b24ea546d"}
Sep 09 20:21:33.337306 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:33.337269    2581 generic.go:356] "Generic (PLEG): container finished" podID="029eacaf-951a-4c17-9a23-6995fe9c5fdc" containerID="44c90e19330e2ec3803af835f3ca3fb394f69f66c8c40e4622e57bcf8a78ebfe" exitCode=0
Sep 09 20:21:33.337712 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:33.337373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484" event={"ID":"029eacaf-951a-4c17-9a23-6995fe9c5fdc","Type":"ContainerDied","Data":"44c90e19330e2ec3803af835f3ca3fb394f69f66c8c40e4622e57bcf8a78ebfe"}
Sep 09 20:21:40.372852 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:40.372813    2581 generic.go:356] "Generic (PLEG): container finished" podID="029eacaf-951a-4c17-9a23-6995fe9c5fdc" containerID="471b1c9a7d892c0b301f2fbd434e74317416af8941f1bfe95ebd22ab2b0a4a08" exitCode=0
Sep 09 20:21:40.373240 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:40.372896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484" event={"ID":"029eacaf-951a-4c17-9a23-6995fe9c5fdc","Type":"ContainerDied","Data":"471b1c9a7d892c0b301f2fbd434e74317416af8941f1bfe95ebd22ab2b0a4a08"}
Sep 09 20:21:41.500416 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:41.500392    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:41.540711 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:41.540681    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv5rh\" (UniqueName: \"kubernetes.io/projected/029eacaf-951a-4c17-9a23-6995fe9c5fdc-kube-api-access-qv5rh\") pod \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") "
Sep 09 20:21:41.540842 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:41.540728    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-bundle\") pod \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") "
Sep 09 20:21:41.540842 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:41.540763    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-util\") pod \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\" (UID: \"029eacaf-951a-4c17-9a23-6995fe9c5fdc\") "
Sep 09 20:21:41.541418 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:41.541393    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-bundle" (OuterVolumeSpecName: "bundle") pod "029eacaf-951a-4c17-9a23-6995fe9c5fdc" (UID: "029eacaf-951a-4c17-9a23-6995fe9c5fdc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:21:41.542964 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:41.542943    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029eacaf-951a-4c17-9a23-6995fe9c5fdc-kube-api-access-qv5rh" (OuterVolumeSpecName: "kube-api-access-qv5rh") pod "029eacaf-951a-4c17-9a23-6995fe9c5fdc" (UID: "029eacaf-951a-4c17-9a23-6995fe9c5fdc"). InnerVolumeSpecName "kube-api-access-qv5rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:21:41.544935 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:41.544909    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-util" (OuterVolumeSpecName: "util") pod "029eacaf-951a-4c17-9a23-6995fe9c5fdc" (UID: "029eacaf-951a-4c17-9a23-6995fe9c5fdc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:21:41.641850 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:41.641759    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-util\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:21:41.641850 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:41.641793    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv5rh\" (UniqueName: \"kubernetes.io/projected/029eacaf-951a-4c17-9a23-6995fe9c5fdc-kube-api-access-qv5rh\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:21:41.641850 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:41.641808    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/029eacaf-951a-4c17-9a23-6995fe9c5fdc-bundle\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:21:42.382482 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:42.382449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484" event={"ID":"029eacaf-951a-4c17-9a23-6995fe9c5fdc","Type":"ContainerDied","Data":"6dcab805a876b7ea88fad7e64c6b5ee17003a4199b76a9c09696c3aa2e8e8f02"}
Sep 09 20:21:42.382482 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:42.382465    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bg484"
Sep 09 20:21:42.382688 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:42.382493    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dcab805a876b7ea88fad7e64c6b5ee17003a4199b76a9c09696c3aa2e8e8f02"
Sep 09 20:21:49.979580 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:49.977444    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng"]
Sep 09 20:21:49.979580 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:49.978060    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029eacaf-951a-4c17-9a23-6995fe9c5fdc" containerName="pull"
Sep 09 20:21:49.979580 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:49.978079    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="029eacaf-951a-4c17-9a23-6995fe9c5fdc" containerName="pull"
Sep 09 20:21:49.979580 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:49.978116    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029eacaf-951a-4c17-9a23-6995fe9c5fdc" containerName="util"
Sep 09 20:21:49.979580 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:49.978124    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="029eacaf-951a-4c17-9a23-6995fe9c5fdc" containerName="util"
Sep 09 20:21:49.979580 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:49.978150    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029eacaf-951a-4c17-9a23-6995fe9c5fdc" containerName="extract"
Sep 09 20:21:49.979580 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:49.978158    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="029eacaf-951a-4c17-9a23-6995fe9c5fdc" containerName="extract"
Sep 09 20:21:49.979580 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:49.978280    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="029eacaf-951a-4c17-9a23-6995fe9c5fdc" containerName="extract"
Sep 09 20:21:49.999048 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:49.999018    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng"]
Sep 09 20:21:49.999232 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:49.999108    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng"
Sep 09 20:21:50.003267 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:50.003243    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:21:50.003431 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:50.003244    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 09 20:21:50.003431 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:50.003274    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7rnm8\""
Sep 09 20:21:50.099652 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:50.099619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czd2x\" (UniqueName: \"kubernetes.io/projected/b917451b-5def-4668-91ba-717afbe71594-kube-api-access-czd2x\") pod \"cert-manager-operator-controller-manager-55c444bf86-b4zng\" (UID: \"b917451b-5def-4668-91ba-717afbe71594\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng"
Sep 09 20:21:50.100005 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:50.099971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b917451b-5def-4668-91ba-717afbe71594-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-b4zng\" (UID: \"b917451b-5def-4668-91ba-717afbe71594\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng"
Sep 09 20:21:50.200492 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:50.200455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b917451b-5def-4668-91ba-717afbe71594-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-b4zng\" (UID: \"b917451b-5def-4668-91ba-717afbe71594\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng"
Sep 09 20:21:50.200703 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:50.200506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czd2x\" (UniqueName: \"kubernetes.io/projected/b917451b-5def-4668-91ba-717afbe71594-kube-api-access-czd2x\") pod \"cert-manager-operator-controller-manager-55c444bf86-b4zng\" (UID: \"b917451b-5def-4668-91ba-717afbe71594\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng"
Sep 09 20:21:50.200806 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:50.200789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b917451b-5def-4668-91ba-717afbe71594-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-b4zng\" (UID: \"b917451b-5def-4668-91ba-717afbe71594\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng"
Sep 09 20:21:50.209061 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:50.209037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czd2x\" (UniqueName: \"kubernetes.io/projected/b917451b-5def-4668-91ba-717afbe71594-kube-api-access-czd2x\") pod \"cert-manager-operator-controller-manager-55c444bf86-b4zng\" (UID: \"b917451b-5def-4668-91ba-717afbe71594\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng"
Sep 09 20:21:50.311176 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:50.311110    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng"
Sep 09 20:21:50.614490 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:50.614462    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng"]
Sep 09 20:21:50.615650 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:21:50.615623    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb917451b_5def_4668_91ba_717afbe71594.slice/crio-28f036f97fbf15a1938393a28c9217fde2b4063eed9ae237b1a544241f12e522 WatchSource:0}: Error finding container 28f036f97fbf15a1938393a28c9217fde2b4063eed9ae237b1a544241f12e522: Status 404 returned error can't find the container with id 28f036f97fbf15a1938393a28c9217fde2b4063eed9ae237b1a544241f12e522
Sep 09 20:21:51.416594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:51.416554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng" event={"ID":"b917451b-5def-4668-91ba-717afbe71594","Type":"ContainerStarted","Data":"28f036f97fbf15a1938393a28c9217fde2b4063eed9ae237b1a544241f12e522"}
Sep 09 20:21:53.429690 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:53.429652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng" event={"ID":"b917451b-5def-4668-91ba-717afbe71594","Type":"ContainerStarted","Data":"a6c2a09b69db5be344cb4ed4627a4cdf1171e0d449086bbc78f0a1e4599ed36c"}
Sep 09 20:21:53.457288 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:53.457240    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-b4zng" podStartSLOduration=2.132333888 podStartE2EDuration="4.457222795s" totalImagesPullingTime="2.324888907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:53.455872302 +0000 UTC m=+334.949083299" watchObservedRunningTime="2026-09-09 20:21:53.457222795 +0000 UTC m=+334.950433778"
Sep 09 20:21:54.211912 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.211882    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5frx4"]
Sep 09 20:21:54.214090 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.214069    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4"
Sep 09 20:21:54.218644 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.218621    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 20:21:54.218950 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.218923    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 20:21:54.229892 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.229870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2brm\" (UniqueName: \"kubernetes.io/projected/03f36af4-15e2-475d-a112-eb1374bbfd4c-kube-api-access-r2brm\") pod \"cert-manager-webhook-6486ffd997-5frx4\" (UID: \"03f36af4-15e2-475d-a112-eb1374bbfd4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4"
Sep 09 20:21:54.229970 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.229914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03f36af4-15e2-475d-a112-eb1374bbfd4c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5frx4\" (UID: \"03f36af4-15e2-475d-a112-eb1374bbfd4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4"
Sep 09 20:21:54.240601 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.240578    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5frx4"]
Sep 09 20:21:54.330454 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.330423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2brm\" (UniqueName: \"kubernetes.io/projected/03f36af4-15e2-475d-a112-eb1374bbfd4c-kube-api-access-r2brm\") pod \"cert-manager-webhook-6486ffd997-5frx4\" (UID: \"03f36af4-15e2-475d-a112-eb1374bbfd4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4"
Sep 09 20:21:54.330621 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.330473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03f36af4-15e2-475d-a112-eb1374bbfd4c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5frx4\" (UID: \"03f36af4-15e2-475d-a112-eb1374bbfd4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4"
Sep 09 20:21:54.346953 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.346917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2brm\" (UniqueName: \"kubernetes.io/projected/03f36af4-15e2-475d-a112-eb1374bbfd4c-kube-api-access-r2brm\") pod \"cert-manager-webhook-6486ffd997-5frx4\" (UID: \"03f36af4-15e2-475d-a112-eb1374bbfd4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4"
Sep 09 20:21:54.347091 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.347072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03f36af4-15e2-475d-a112-eb1374bbfd4c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5frx4\" (UID: \"03f36af4-15e2-475d-a112-eb1374bbfd4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4"
Sep 09 20:21:54.418819 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.418785    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"]
Sep 09 20:21:54.421009 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.420993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:21:54.423600 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.423578    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mwfck\""
Sep 09 20:21:54.423600 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.423596    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:21:54.423732 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.423597    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:21:54.431146 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.431124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8fw\" (UniqueName: \"kubernetes.io/projected/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-kube-api-access-pg8fw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:21:54.431453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.431174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:21:54.431453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.431225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:21:54.436703 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.436678    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"]
Sep 09 20:21:54.531637 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.531549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:21:54.531637 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.531618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8fw\" (UniqueName: \"kubernetes.io/projected/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-kube-api-access-pg8fw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:21:54.531841 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.531664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:21:54.532076 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.532056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:21:54.532115 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.532086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:21:54.533442 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.533425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4"
Sep 09 20:21:54.541337 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.541313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8fw\" (UniqueName: \"kubernetes.io/projected/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-kube-api-access-pg8fw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:21:54.730520 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.730483    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:21:54.809982 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:54.809958    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5frx4"]
Sep 09 20:21:54.813026 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:21:54.812999    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03f36af4_15e2_475d_a112_eb1374bbfd4c.slice/crio-e23592e3b252ea2c3d4f6859f5e8ffe92c6889dc23fbe676e8b75684793ca2c7 WatchSource:0}: Error finding container e23592e3b252ea2c3d4f6859f5e8ffe92c6889dc23fbe676e8b75684793ca2c7: Status 404 returned error can't find the container with id e23592e3b252ea2c3d4f6859f5e8ffe92c6889dc23fbe676e8b75684793ca2c7
Sep 09 20:21:55.011698 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:55.011675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"]
Sep 09 20:21:55.013893 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:21:55.013868    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef05de9_bbf1_4b9c_bee0_10bb9f9f076d.slice/crio-8adc827f7ec0630ffffc2559e61c550b8eb8974e99ef6f2ba0efd8eddbc306c7 WatchSource:0}: Error finding container 8adc827f7ec0630ffffc2559e61c550b8eb8974e99ef6f2ba0efd8eddbc306c7: Status 404 returned error can't find the container with id 8adc827f7ec0630ffffc2559e61c550b8eb8974e99ef6f2ba0efd8eddbc306c7
Sep 09 20:21:55.440375 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:55.440327    2581 generic.go:356] "Generic (PLEG): container finished" podID="fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" containerID="8087d7ee5f07aa2c5db236c32512573ea49615206f945bb8a5c9d20913e543f8" exitCode=0
Sep 09 20:21:55.440787 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:55.440414    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz" event={"ID":"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d","Type":"ContainerDied","Data":"8087d7ee5f07aa2c5db236c32512573ea49615206f945bb8a5c9d20913e543f8"}
Sep 09 20:21:55.440787 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:55.440480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz" event={"ID":"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d","Type":"ContainerStarted","Data":"8adc827f7ec0630ffffc2559e61c550b8eb8974e99ef6f2ba0efd8eddbc306c7"}
Sep 09 20:21:55.441719 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:55.441688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4" event={"ID":"03f36af4-15e2-475d-a112-eb1374bbfd4c","Type":"ContainerStarted","Data":"e23592e3b252ea2c3d4f6859f5e8ffe92c6889dc23fbe676e8b75684793ca2c7"}
Sep 09 20:21:59.460734 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:59.460694    2581 generic.go:356] "Generic (PLEG): container finished" podID="fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" containerID="d036cc7dbb7031a2dacf79b6bbd703889e749efab221e850a6a8be6a716306f4" exitCode=0
Sep 09 20:21:59.461190 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:59.460781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz" event={"ID":"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d","Type":"ContainerDied","Data":"d036cc7dbb7031a2dacf79b6bbd703889e749efab221e850a6a8be6a716306f4"}
Sep 09 20:21:59.462503 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:59.462478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4" event={"ID":"03f36af4-15e2-475d-a112-eb1374bbfd4c","Type":"ContainerStarted","Data":"f08344bf328bf4b4a437fd282b8487893a1a9f2bfdc7bab37b184bf0b904351c"}
Sep 09 20:21:59.462607 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:59.462559    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4"
Sep 09 20:21:59.543115 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:21:59.543070    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4" podStartSLOduration=1.8874690950000002 podStartE2EDuration="5.543050345s" totalImagesPullingTime="3.65558125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:21:59.542935122 +0000 UTC m=+341.036146106" watchObservedRunningTime="2026-09-09 20:21:59.543050345 +0000 UTC m=+341.036261330"
Sep 09 20:22:00.468962 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:00.468932    2581 generic.go:356] "Generic (PLEG): container finished" podID="fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" containerID="eb7c8658bf06622bbad4d56dafea59bc0bb4611956275c99bb593a2f638836b5" exitCode=0
Sep 09 20:22:00.469361 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:00.469027    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz" event={"ID":"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d","Type":"ContainerDied","Data":"eb7c8658bf06622bbad4d56dafea59bc0bb4611956275c99bb593a2f638836b5"}
Sep 09 20:22:01.591727 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:01.591699    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:22:01.686163 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:01.686129    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-util\") pod \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") "
Sep 09 20:22:01.686321 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:01.686176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg8fw\" (UniqueName: \"kubernetes.io/projected/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-kube-api-access-pg8fw\") pod \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") "
Sep 09 20:22:01.686321 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:01.686200    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-bundle\") pod \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\" (UID: \"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d\") "
Sep 09 20:22:01.686599 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:01.686570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-bundle" (OuterVolumeSpecName: "bundle") pod "fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" (UID: "fef05de9-bbf1-4b9c-bee0-10bb9f9f076d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:01.688322 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:01.688299    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-kube-api-access-pg8fw" (OuterVolumeSpecName: "kube-api-access-pg8fw") pod "fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" (UID: "fef05de9-bbf1-4b9c-bee0-10bb9f9f076d"). InnerVolumeSpecName "kube-api-access-pg8fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:01.690666 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:01.690644    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-util" (OuterVolumeSpecName: "util") pod "fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" (UID: "fef05de9-bbf1-4b9c-bee0-10bb9f9f076d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:01.787718 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:01.787646    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-util\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:22:01.787718 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:01.787670    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg8fw\" (UniqueName: \"kubernetes.io/projected/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-kube-api-access-pg8fw\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:22:01.787718 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:01.787681    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fef05de9-bbf1-4b9c-bee0-10bb9f9f076d-bundle\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:22:02.479149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:02.479116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz" event={"ID":"fef05de9-bbf1-4b9c-bee0-10bb9f9f076d","Type":"ContainerDied","Data":"8adc827f7ec0630ffffc2559e61c550b8eb8974e99ef6f2ba0efd8eddbc306c7"}
Sep 09 20:22:02.479320 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:02.479164    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8adc827f7ec0630ffffc2559e61c550b8eb8974e99ef6f2ba0efd8eddbc306c7"
Sep 09 20:22:02.479320 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:02.479166    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpdfmz"
Sep 09 20:22:05.471307 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:05.471273    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5frx4"
Sep 09 20:22:07.871106 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.871063    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6"]
Sep 09 20:22:07.871504 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.871360    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" containerName="pull"
Sep 09 20:22:07.871504 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.871373    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" containerName="pull"
Sep 09 20:22:07.871504 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.871391    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" containerName="util"
Sep 09 20:22:07.871504 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.871396    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" containerName="util"
Sep 09 20:22:07.871504 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.871404    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" containerName="extract"
Sep 09 20:22:07.871504 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.871409    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" containerName="extract"
Sep 09 20:22:07.871504 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.871460    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fef05de9-bbf1-4b9c-bee0-10bb9f9f076d" containerName="extract"
Sep 09 20:22:07.875002 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.874986    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6"
Sep 09 20:22:07.877766 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.877744    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:22:07.878702 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.878683    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 20:22:07.878797 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.878693    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-8qhcg\""
Sep 09 20:22:07.884317 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.884294    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6"]
Sep 09 20:22:07.930060 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.930023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0345558e-e785-4c86-86d1-63b79652d66e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mvmv6\" (UID: \"0345558e-e785-4c86-86d1-63b79652d66e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6"
Sep 09 20:22:07.930060 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:07.930060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phzlg\" (UniqueName: \"kubernetes.io/projected/0345558e-e785-4c86-86d1-63b79652d66e-kube-api-access-phzlg\") pod \"openshift-lws-operator-fd8ccff4c-mvmv6\" (UID: \"0345558e-e785-4c86-86d1-63b79652d66e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6"
Sep 09 20:22:08.030501 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:08.030464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0345558e-e785-4c86-86d1-63b79652d66e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mvmv6\" (UID: \"0345558e-e785-4c86-86d1-63b79652d66e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6"
Sep 09 20:22:08.030748 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:08.030507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzlg\" (UniqueName: \"kubernetes.io/projected/0345558e-e785-4c86-86d1-63b79652d66e-kube-api-access-phzlg\") pod \"openshift-lws-operator-fd8ccff4c-mvmv6\" (UID: \"0345558e-e785-4c86-86d1-63b79652d66e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6"
Sep 09 20:22:08.030849 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:08.030828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0345558e-e785-4c86-86d1-63b79652d66e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mvmv6\" (UID: \"0345558e-e785-4c86-86d1-63b79652d66e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6"
Sep 09 20:22:08.041985 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:08.041959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phzlg\" (UniqueName: \"kubernetes.io/projected/0345558e-e785-4c86-86d1-63b79652d66e-kube-api-access-phzlg\") pod \"openshift-lws-operator-fd8ccff4c-mvmv6\" (UID: \"0345558e-e785-4c86-86d1-63b79652d66e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6"
Sep 09 20:22:08.184245 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:08.184222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6"
Sep 09 20:22:08.458831 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:08.458802    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6"]
Sep 09 20:22:08.461431 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:22:08.461397    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0345558e_e785_4c86_86d1_63b79652d66e.slice/crio-74c03325ba8ba728737e34de774b0bf200e23b23aa228f688ebeefab29611239 WatchSource:0}: Error finding container 74c03325ba8ba728737e34de774b0bf200e23b23aa228f688ebeefab29611239: Status 404 returned error can't find the container with id 74c03325ba8ba728737e34de774b0bf200e23b23aa228f688ebeefab29611239
Sep 09 20:22:08.506173 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:08.506138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6" event={"ID":"0345558e-e785-4c86-86d1-63b79652d66e","Type":"ContainerStarted","Data":"74c03325ba8ba728737e34de774b0bf200e23b23aa228f688ebeefab29611239"}
Sep 09 20:22:12.526938 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:12.526903    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6" event={"ID":"0345558e-e785-4c86-86d1-63b79652d66e","Type":"ContainerStarted","Data":"bab7dad05df4a357aac8210dff2db2bee4f735ff1d41e262e73a0b61d47f9dc0"}
Sep 09 20:22:12.544355 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:12.544303    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mvmv6" podStartSLOduration=2.492486348 podStartE2EDuration="5.544288265s" totalImagesPullingTime="3.051801917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:12.542977282 +0000 UTC m=+354.036188268" watchObservedRunningTime="2026-09-09 20:22:12.544288265 +0000 UTC m=+354.037499248"
Sep 09 20:22:15.073819 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.073787    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"]
Sep 09 20:22:15.077183 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.077167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:15.079818 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.079795    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:22:15.080809 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.080792    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:22:15.080878 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.080790    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mwfck\""
Sep 09 20:22:15.086125 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.086101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"]
Sep 09 20:22:15.187105 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.187074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8r7k\" (UniqueName: \"kubernetes.io/projected/846945ad-e284-47d5-ad22-1beb84e87e98-kube-api-access-j8r7k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:15.187256 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.187129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:15.187256 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.187150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:15.287610 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.287573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:15.287610 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.287614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:15.287784 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.287662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8r7k\" (UniqueName: \"kubernetes.io/projected/846945ad-e284-47d5-ad22-1beb84e87e98-kube-api-access-j8r7k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:15.287936 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.287918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:15.288024 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.288007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:15.298567 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.298543    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8r7k\" (UniqueName: \"kubernetes.io/projected/846945ad-e284-47d5-ad22-1beb84e87e98-kube-api-access-j8r7k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:15.386453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.386373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:15.659175 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:15.659150    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"]
Sep 09 20:22:15.661559 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:22:15.661532    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846945ad_e284_47d5_ad22_1beb84e87e98.slice/crio-8cb3fa576ce15cbcfebefb611c4a199c444f0c7cdeebea1f7c52169274c93923 WatchSource:0}: Error finding container 8cb3fa576ce15cbcfebefb611c4a199c444f0c7cdeebea1f7c52169274c93923: Status 404 returned error can't find the container with id 8cb3fa576ce15cbcfebefb611c4a199c444f0c7cdeebea1f7c52169274c93923
Sep 09 20:22:16.544877 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:16.544841    2581 generic.go:356] "Generic (PLEG): container finished" podID="846945ad-e284-47d5-ad22-1beb84e87e98" containerID="b8061c6fa5e478282dd2bb4d90a0333822d08f033cb77487431f2a91f1999d26" exitCode=0
Sep 09 20:22:16.545253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:16.544897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs" event={"ID":"846945ad-e284-47d5-ad22-1beb84e87e98","Type":"ContainerDied","Data":"b8061c6fa5e478282dd2bb4d90a0333822d08f033cb77487431f2a91f1999d26"}
Sep 09 20:22:16.545253 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:16.544939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs" event={"ID":"846945ad-e284-47d5-ad22-1beb84e87e98","Type":"ContainerStarted","Data":"8cb3fa576ce15cbcfebefb611c4a199c444f0c7cdeebea1f7c52169274c93923"}
Sep 09 20:22:17.551653 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:17.551561    2581 generic.go:356] "Generic (PLEG): container finished" podID="846945ad-e284-47d5-ad22-1beb84e87e98" containerID="a8a2560d5920fcca417002768df0896a62a4603b9c8475eac83b349377fa808a" exitCode=0
Sep 09 20:22:17.552038 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:17.551652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs" event={"ID":"846945ad-e284-47d5-ad22-1beb84e87e98","Type":"ContainerDied","Data":"a8a2560d5920fcca417002768df0896a62a4603b9c8475eac83b349377fa808a"}
Sep 09 20:22:18.557729 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:18.557694    2581 generic.go:356] "Generic (PLEG): container finished" podID="846945ad-e284-47d5-ad22-1beb84e87e98" containerID="18b9685f81010bb370c9a7b74567921ec50c56634a3b43d28ab1e16a1b61414a" exitCode=0
Sep 09 20:22:18.558105 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:18.557775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs" event={"ID":"846945ad-e284-47d5-ad22-1beb84e87e98","Type":"ContainerDied","Data":"18b9685f81010bb370c9a7b74567921ec50c56634a3b43d28ab1e16a1b61414a"}
Sep 09 20:22:19.687531 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:19.687505    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:19.726250 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:19.726212    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-util\") pod \"846945ad-e284-47d5-ad22-1beb84e87e98\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") "
Sep 09 20:22:19.726431 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:19.726270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8r7k\" (UniqueName: \"kubernetes.io/projected/846945ad-e284-47d5-ad22-1beb84e87e98-kube-api-access-j8r7k\") pod \"846945ad-e284-47d5-ad22-1beb84e87e98\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") "
Sep 09 20:22:19.726431 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:19.726295    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-bundle\") pod \"846945ad-e284-47d5-ad22-1beb84e87e98\" (UID: \"846945ad-e284-47d5-ad22-1beb84e87e98\") "
Sep 09 20:22:19.727281 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:19.727250    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-bundle" (OuterVolumeSpecName: "bundle") pod "846945ad-e284-47d5-ad22-1beb84e87e98" (UID: "846945ad-e284-47d5-ad22-1beb84e87e98"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:19.728544 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:19.728518    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846945ad-e284-47d5-ad22-1beb84e87e98-kube-api-access-j8r7k" (OuterVolumeSpecName: "kube-api-access-j8r7k") pod "846945ad-e284-47d5-ad22-1beb84e87e98" (UID: "846945ad-e284-47d5-ad22-1beb84e87e98"). InnerVolumeSpecName "kube-api-access-j8r7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:19.734010 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:19.733982    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-util" (OuterVolumeSpecName: "util") pod "846945ad-e284-47d5-ad22-1beb84e87e98" (UID: "846945ad-e284-47d5-ad22-1beb84e87e98"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:19.827839 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:19.827755    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8r7k\" (UniqueName: \"kubernetes.io/projected/846945ad-e284-47d5-ad22-1beb84e87e98-kube-api-access-j8r7k\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:22:19.827839 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:19.827785    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-bundle\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:22:19.827839 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:19.827795    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/846945ad-e284-47d5-ad22-1beb84e87e98-util\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:22:20.568221 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:20.568191    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs"
Sep 09 20:22:20.568421 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:20.568194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afskrqs" event={"ID":"846945ad-e284-47d5-ad22-1beb84e87e98","Type":"ContainerDied","Data":"8cb3fa576ce15cbcfebefb611c4a199c444f0c7cdeebea1f7c52169274c93923"}
Sep 09 20:22:20.568421 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:20.568369    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cb3fa576ce15cbcfebefb611c4a199c444f0c7cdeebea1f7c52169274c93923"
Sep 09 20:22:25.246156 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.246120    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"]
Sep 09 20:22:25.246635 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.246415    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846945ad-e284-47d5-ad22-1beb84e87e98" containerName="pull"
Sep 09 20:22:25.246635 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.246427    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="846945ad-e284-47d5-ad22-1beb84e87e98" containerName="pull"
Sep 09 20:22:25.246635 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.246439    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846945ad-e284-47d5-ad22-1beb84e87e98" containerName="extract"
Sep 09 20:22:25.246635 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.246445    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="846945ad-e284-47d5-ad22-1beb84e87e98" containerName="extract"
Sep 09 20:22:25.246635 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.246453    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846945ad-e284-47d5-ad22-1beb84e87e98" containerName="util"
Sep 09 20:22:25.246635 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.246459    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="846945ad-e284-47d5-ad22-1beb84e87e98" containerName="util"
Sep 09 20:22:25.246635 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.246512    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="846945ad-e284-47d5-ad22-1beb84e87e98" containerName="extract"
Sep 09 20:22:25.252455 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.252437    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:25.255129 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.255103    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:22:25.255248 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.255207    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mwfck\""
Sep 09 20:22:25.256437 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.256408    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:22:25.258584 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.258561    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"]
Sep 09 20:22:25.369509 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.369468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:25.369694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.369525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7d9\" (UniqueName: \"kubernetes.io/projected/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-kube-api-access-5k7d9\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:25.369694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.369620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:25.470050 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.470016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:25.470225 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.470064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7d9\" (UniqueName: \"kubernetes.io/projected/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-kube-api-access-5k7d9\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:25.470225 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.470106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:25.470507 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.470486    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:25.470561 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.470509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:25.483443 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.483413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7d9\" (UniqueName: \"kubernetes.io/projected/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-kube-api-access-5k7d9\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:25.562463 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.562384    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:25.844717 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:25.844692    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"]
Sep 09 20:22:25.847137 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:22:25.847110    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81aef59d_fb02_4aa0_a31c_46b44b80e6c7.slice/crio-dc86ce75a0bed4e6325da49c69bd86bbc5e7cc73831cbc8f0e7b1c81c538e4dc WatchSource:0}: Error finding container dc86ce75a0bed4e6325da49c69bd86bbc5e7cc73831cbc8f0e7b1c81c538e4dc: Status 404 returned error can't find the container with id dc86ce75a0bed4e6325da49c69bd86bbc5e7cc73831cbc8f0e7b1c81c538e4dc
Sep 09 20:22:26.595076 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:26.595034    2581 generic.go:356] "Generic (PLEG): container finished" podID="81aef59d-fb02-4aa0-a31c-46b44b80e6c7" containerID="7a1709a1875d3cd02c5ccf52c83a9fa20e0d17c5959bbaef3f06c7ed9f4b3560" exitCode=0
Sep 09 20:22:26.595515 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:26.595103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx" event={"ID":"81aef59d-fb02-4aa0-a31c-46b44b80e6c7","Type":"ContainerDied","Data":"7a1709a1875d3cd02c5ccf52c83a9fa20e0d17c5959bbaef3f06c7ed9f4b3560"}
Sep 09 20:22:26.595515 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:26.595166    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx" event={"ID":"81aef59d-fb02-4aa0-a31c-46b44b80e6c7","Type":"ContainerStarted","Data":"dc86ce75a0bed4e6325da49c69bd86bbc5e7cc73831cbc8f0e7b1c81c538e4dc"}
Sep 09 20:22:27.598456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.598426    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"]
Sep 09 20:22:27.602133 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.602111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx" event={"ID":"81aef59d-fb02-4aa0-a31c-46b44b80e6c7","Type":"ContainerStarted","Data":"c33d27d8e35e4baeac3eff94809fc9a8b5fe77fc14d53aa457d718ac71769500"}
Sep 09 20:22:27.602234 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.602189    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.607057 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.607036    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 20:22:27.607279 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.607262    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 20:22:27.607396 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.607281    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 20:22:27.607396 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.607294    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p75md\""
Sep 09 20:22:27.607396 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.607328    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 20:22:27.626861 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.626837    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"]
Sep 09 20:22:27.688764 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.688733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d171978-73e5-4a61-9eaf-8ff6047520fb-manifests\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.688931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.688777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d171978-73e5-4a61-9eaf-8ff6047520fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.688931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.688870    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d171978-73e5-4a61-9eaf-8ff6047520fb-tmp\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.688931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.688909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d171978-73e5-4a61-9eaf-8ff6047520fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.689069 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.688939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4gt7\" (UniqueName: \"kubernetes.io/projected/3d171978-73e5-4a61-9eaf-8ff6047520fb-kube-api-access-l4gt7\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.790310 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.790216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d171978-73e5-4a61-9eaf-8ff6047520fb-tmp\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.790310 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.790262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d171978-73e5-4a61-9eaf-8ff6047520fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.790310 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.790288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4gt7\" (UniqueName: \"kubernetes.io/projected/3d171978-73e5-4a61-9eaf-8ff6047520fb-kube-api-access-l4gt7\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.790603 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.790337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d171978-73e5-4a61-9eaf-8ff6047520fb-manifests\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.790603 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.790413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d171978-73e5-4a61-9eaf-8ff6047520fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.790737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.790719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d171978-73e5-4a61-9eaf-8ff6047520fb-manifests\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.792660 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.792630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d171978-73e5-4a61-9eaf-8ff6047520fb-tmp\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.792889 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.792869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d171978-73e5-4a61-9eaf-8ff6047520fb-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.792959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.792907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d171978-73e5-4a61-9eaf-8ff6047520fb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.811334 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.811312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4gt7\" (UniqueName: \"kubernetes.io/projected/3d171978-73e5-4a61-9eaf-8ff6047520fb-kube-api-access-l4gt7\") pod \"opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n\" (UID: \"3d171978-73e5-4a61-9eaf-8ff6047520fb\") " pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:27.958967 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:27.958932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:28.256250 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:28.256226    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"]
Sep 09 20:22:28.257484 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:22:28.257457    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d171978_73e5_4a61_9eaf_8ff6047520fb.slice/crio-0bff498977faee682ae29478de752d64b463186759b24b9c00856885c206c8ff WatchSource:0}: Error finding container 0bff498977faee682ae29478de752d64b463186759b24b9c00856885c206c8ff: Status 404 returned error can't find the container with id 0bff498977faee682ae29478de752d64b463186759b24b9c00856885c206c8ff
Sep 09 20:22:28.607915 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:28.607821    2581 generic.go:356] "Generic (PLEG): container finished" podID="81aef59d-fb02-4aa0-a31c-46b44b80e6c7" containerID="c33d27d8e35e4baeac3eff94809fc9a8b5fe77fc14d53aa457d718ac71769500" exitCode=0
Sep 09 20:22:28.607915 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:28.607908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx" event={"ID":"81aef59d-fb02-4aa0-a31c-46b44b80e6c7","Type":"ContainerDied","Data":"c33d27d8e35e4baeac3eff94809fc9a8b5fe77fc14d53aa457d718ac71769500"}
Sep 09 20:22:28.609315 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:28.609294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n" event={"ID":"3d171978-73e5-4a61-9eaf-8ff6047520fb","Type":"ContainerStarted","Data":"0bff498977faee682ae29478de752d64b463186759b24b9c00856885c206c8ff"}
Sep 09 20:22:29.617640 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:29.617600    2581 generic.go:356] "Generic (PLEG): container finished" podID="81aef59d-fb02-4aa0-a31c-46b44b80e6c7" containerID="ff7d2a8c85fd3f3d6c80f930a16b3e3e896ff9365d5b929ab9d6dd3bd43d2010" exitCode=0
Sep 09 20:22:29.618065 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:29.617666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx" event={"ID":"81aef59d-fb02-4aa0-a31c-46b44b80e6c7","Type":"ContainerDied","Data":"ff7d2a8c85fd3f3d6c80f930a16b3e3e896ff9365d5b929ab9d6dd3bd43d2010"}
Sep 09 20:22:31.427148 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.427126    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:31.524503 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.524425    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k7d9\" (UniqueName: \"kubernetes.io/projected/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-kube-api-access-5k7d9\") pod \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") "
Sep 09 20:22:31.524619 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.524523    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-util\") pod \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") "
Sep 09 20:22:31.524619 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.524569    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-bundle\") pod \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\" (UID: \"81aef59d-fb02-4aa0-a31c-46b44b80e6c7\") "
Sep 09 20:22:31.525479 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.525448    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-bundle" (OuterVolumeSpecName: "bundle") pod "81aef59d-fb02-4aa0-a31c-46b44b80e6c7" (UID: "81aef59d-fb02-4aa0-a31c-46b44b80e6c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:31.527168 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.527143    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-kube-api-access-5k7d9" (OuterVolumeSpecName: "kube-api-access-5k7d9") pod "81aef59d-fb02-4aa0-a31c-46b44b80e6c7" (UID: "81aef59d-fb02-4aa0-a31c-46b44b80e6c7"). InnerVolumeSpecName "kube-api-access-5k7d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:31.533041 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.533013    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-util" (OuterVolumeSpecName: "util") pod "81aef59d-fb02-4aa0-a31c-46b44b80e6c7" (UID: "81aef59d-fb02-4aa0-a31c-46b44b80e6c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:22:31.625174 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.625138    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-util\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:22:31.625174 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.625169    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-bundle\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:22:31.625174 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.625180    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k7d9\" (UniqueName: \"kubernetes.io/projected/81aef59d-fb02-4aa0-a31c-46b44b80e6c7-kube-api-access-5k7d9\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:22:31.630012 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.629990    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx"
Sep 09 20:22:31.630134 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.629988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hj5dx" event={"ID":"81aef59d-fb02-4aa0-a31c-46b44b80e6c7","Type":"ContainerDied","Data":"dc86ce75a0bed4e6325da49c69bd86bbc5e7cc73831cbc8f0e7b1c81c538e4dc"}
Sep 09 20:22:31.630190 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.630134    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc86ce75a0bed4e6325da49c69bd86bbc5e7cc73831cbc8f0e7b1c81c538e4dc"
Sep 09 20:22:31.631857 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:31.631834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n" event={"ID":"3d171978-73e5-4a61-9eaf-8ff6047520fb","Type":"ContainerStarted","Data":"e4c6d7e9b98faad69fbd5052fd65147aa93143fb57249fea23a242d907709d0b"}
Sep 09 20:22:32.638088 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:32.638047    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d171978-73e5-4a61-9eaf-8ff6047520fb" containerID="e4c6d7e9b98faad69fbd5052fd65147aa93143fb57249fea23a242d907709d0b" exitCode=0
Sep 09 20:22:32.638493 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:32.638104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n" event={"ID":"3d171978-73e5-4a61-9eaf-8ff6047520fb","Type":"ContainerDied","Data":"e4c6d7e9b98faad69fbd5052fd65147aa93143fb57249fea23a242d907709d0b"}
Sep 09 20:22:33.645910 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:33.645875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n" event={"ID":"3d171978-73e5-4a61-9eaf-8ff6047520fb","Type":"ContainerStarted","Data":"f1bc19c3ca0d01ed569cc5d2f64853fa67e7bfeb74c8ec2fdee5035823cd3bda"}
Sep 09 20:22:33.646284 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:33.645976    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:33.670409 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:33.670362    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n" podStartSLOduration=2.098273603 podStartE2EDuration="6.670329682s" totalImagesPullingTime="3.559251735s" totalInitContainerRuntime="1.012804344s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:33.6672549 +0000 UTC m=+375.160465909" watchObservedRunningTime="2026-09-09 20:22:33.670329682 +0000 UTC m=+375.163540666"
Sep 09 20:22:44.651984 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:44.651956    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n"
Sep 09 20:22:47.064838 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.064804    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-758cfc744b-627r6"]
Sep 09 20:22:47.065261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.065096    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81aef59d-fb02-4aa0-a31c-46b44b80e6c7" containerName="util"
Sep 09 20:22:47.065261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.065106    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="81aef59d-fb02-4aa0-a31c-46b44b80e6c7" containerName="util"
Sep 09 20:22:47.065261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.065118    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81aef59d-fb02-4aa0-a31c-46b44b80e6c7" containerName="pull"
Sep 09 20:22:47.065261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.065123    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="81aef59d-fb02-4aa0-a31c-46b44b80e6c7" containerName="pull"
Sep 09 20:22:47.065261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.065140    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81aef59d-fb02-4aa0-a31c-46b44b80e6c7" containerName="extract"
Sep 09 20:22:47.065261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.065145    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="81aef59d-fb02-4aa0-a31c-46b44b80e6c7" containerName="extract"
Sep 09 20:22:47.065261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.065196    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="81aef59d-fb02-4aa0-a31c-46b44b80e6c7" containerName="extract"
Sep 09 20:22:47.068800 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.068784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:47.072849 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.072820    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 20:22:47.073186 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.073171    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 20:22:47.073922 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.073903    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fhwmj\""
Sep 09 20:22:47.117959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.117918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-758cfc744b-627r6"]
Sep 09 20:22:47.147161 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.147120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-tmp\") pod \"kube-auth-proxy-758cfc744b-627r6\" (UID: \"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:47.147364 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.147172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vtx8\" (UniqueName: \"kubernetes.io/projected/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-kube-api-access-4vtx8\") pod \"kube-auth-proxy-758cfc744b-627r6\" (UID: \"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:47.147364 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.147205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-tls-certs\") pod \"kube-auth-proxy-758cfc744b-627r6\" (UID: \"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:47.248285 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.248243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-tmp\") pod \"kube-auth-proxy-758cfc744b-627r6\" (UID: \"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:47.248512 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.248327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vtx8\" (UniqueName: \"kubernetes.io/projected/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-kube-api-access-4vtx8\") pod \"kube-auth-proxy-758cfc744b-627r6\" (UID: \"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:47.248512 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.248392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-tls-certs\") pod \"kube-auth-proxy-758cfc744b-627r6\" (UID: \"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:47.248512 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:22:47.248483    2581 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 09 20:22:47.248691 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:22:47.248571    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-tls-certs podName:d14e73dc-c91b-420c-8d6e-d59c9bb38ff0 nodeName:}" failed. No retries permitted until 2026-09-09 20:22:47.748550888 +0000 UTC m=+389.241761854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-tls-certs") pod "kube-auth-proxy-758cfc744b-627r6" (UID: "d14e73dc-c91b-420c-8d6e-d59c9bb38ff0") : secret "kube-auth-proxy-tls" not found
Sep 09 20:22:47.250792 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.250768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-tmp\") pod \"kube-auth-proxy-758cfc744b-627r6\" (UID: \"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:47.285090 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.285060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vtx8\" (UniqueName: \"kubernetes.io/projected/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-kube-api-access-4vtx8\") pod \"kube-auth-proxy-758cfc744b-627r6\" (UID: \"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:47.754291 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.754247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-tls-certs\") pod \"kube-auth-proxy-758cfc744b-627r6\" (UID: \"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:47.756870 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.756843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d14e73dc-c91b-420c-8d6e-d59c9bb38ff0-tls-certs\") pod \"kube-auth-proxy-758cfc744b-627r6\" (UID: \"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0\") " pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:47.978481 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:47.978441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6"
Sep 09 20:22:48.329534 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:48.329508    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-758cfc744b-627r6"]
Sep 09 20:22:48.331929 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:22:48.331903    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14e73dc_c91b_420c_8d6e_d59c9bb38ff0.slice/crio-2259eb42e27909be5888da5c4116822d8e559fb58be4b15e4f7fbf9d8b1010ef WatchSource:0}: Error finding container 2259eb42e27909be5888da5c4116822d8e559fb58be4b15e4f7fbf9d8b1010ef: Status 404 returned error can't find the container with id 2259eb42e27909be5888da5c4116822d8e559fb58be4b15e4f7fbf9d8b1010ef
Sep 09 20:22:48.714976 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:48.714938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6" event={"ID":"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0","Type":"ContainerStarted","Data":"2259eb42e27909be5888da5c4116822d8e559fb58be4b15e4f7fbf9d8b1010ef"}
Sep 09 20:22:49.129517 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.128989    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"]
Sep 09 20:22:49.133076 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.132941    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.136551 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.136528    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 20:22:49.136819 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.136802    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:22:49.137289 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.137079    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 20:22:49.137289 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.137097    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 20:22:49.137465 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.137329    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4srjj\""
Sep 09 20:22:49.157727 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.157390    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"]
Sep 09 20:22:49.269536 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.269494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05d7159-1654-4248-97c7-6af9ca22a523-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.269730 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.269629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.269730 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.269667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.269730 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.269693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.269730 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.269723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.269926 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.269777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plf7b\" (UniqueName: \"kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-kube-api-access-plf7b\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.269926 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.269820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.370775 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.370734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plf7b\" (UniqueName: \"kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-kube-api-access-plf7b\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.371249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.370797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.371249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.370852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05d7159-1654-4248-97c7-6af9ca22a523-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.371249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.371021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.371249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.371062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.371249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.371087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.371249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.371117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.371853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.371789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.374414 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.374366    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.374638 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.374616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.374872 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.374849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05d7159-1654-4248-97c7-6af9ca22a523-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.375869 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.375847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.380709 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.380639    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.381609 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.381565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plf7b\" (UniqueName: \"kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-kube-api-access-plf7b\") pod \"istiod-openshift-gateway-7b7995d5cd-rfx7d\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.446572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.446538    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:49.745693 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:49.745666    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"]
Sep 09 20:22:51.021297 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:22:51.021266    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05d7159_1654_4248_97c7_6af9ca22a523.slice/crio-a796bf22541b3e0f8a03ea4182e7b38dcbea071a189ccaf7f32ff165835f2aba WatchSource:0}: Error finding container a796bf22541b3e0f8a03ea4182e7b38dcbea071a189ccaf7f32ff165835f2aba: Status 404 returned error can't find the container with id a796bf22541b3e0f8a03ea4182e7b38dcbea071a189ccaf7f32ff165835f2aba
Sep 09 20:22:51.731016 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:51.730973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d" event={"ID":"b05d7159-1654-4248-97c7-6af9ca22a523","Type":"ContainerStarted","Data":"a796bf22541b3e0f8a03ea4182e7b38dcbea071a189ccaf7f32ff165835f2aba"}
Sep 09 20:22:51.732432 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:51.732407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6" event={"ID":"d14e73dc-c91b-420c-8d6e-d59c9bb38ff0","Type":"ContainerStarted","Data":"9d70a3547ad94428316e94841e5cdd73fc7d3520cb0c034e4711db829d5594e2"}
Sep 09 20:22:51.749873 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:51.749799    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-758cfc744b-627r6" podStartSLOduration=2.009432532 podStartE2EDuration="4.749780151s" totalImagesPullingTime="2.740347619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:51.74887735 +0000 UTC m=+393.242088333" watchObservedRunningTime="2026-09-09 20:22:51.749780151 +0000 UTC m=+393.242991137"
Sep 09 20:22:53.586011 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:53.585970    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 20:22:53.586321 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:53.586041    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 09 20:22:53.744964 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:53.744927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d" event={"ID":"b05d7159-1654-4248-97c7-6af9ca22a523","Type":"ContainerStarted","Data":"70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934"}
Sep 09 20:22:53.745167 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:53.745018    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:53.771097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:53.771033    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d" podStartSLOduration=2.209073009 podStartE2EDuration="4.771010056s" totalImagesPullingTime="2.561937047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:53.76888263 +0000 UTC m=+395.262093613" watchObservedRunningTime="2026-09-09 20:22:53.771010056 +0000 UTC m=+395.264221042"
Sep 09 20:22:54.751685 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:54.751652    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-rfx7d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:22:54.752128 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:54.751710    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d" podUID="b05d7159-1654-4248-97c7-6af9ca22a523" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:22:57.751237 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:57.751205    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:22:57.843452 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:57.843420    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c78ccbc66-dlwks"]
Sep 09 20:22:57.848332 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:57.848303    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:57.864955 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:57.864926    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c78ccbc66-dlwks"]
Sep 09 20:22:57.954058 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:57.954019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnbrd\" (UniqueName: \"kubernetes.io/projected/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-kube-api-access-lnbrd\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:57.954058 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:57.954061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-oauth-serving-cert\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:57.954261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:57.954164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-service-ca\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:57.954261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:57.954230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-console-serving-cert\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:57.954378 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:57.954271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-console-config\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:57.954378 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:57.954323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-trusted-ca-bundle\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:57.954458 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:57.954380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-console-oauth-config\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.054857 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.054768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-service-ca\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.054857 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.054815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-console-serving-cert\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.054857 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.054838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-console-config\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.055154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.055066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-trusted-ca-bundle\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.055154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.055113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-console-oauth-config\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.055257 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.055155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnbrd\" (UniqueName: \"kubernetes.io/projected/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-kube-api-access-lnbrd\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.055257 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.055184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-oauth-serving-cert\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.055721 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.055698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-service-ca\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.055849 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.055827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-oauth-serving-cert\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.055928 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.055854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-console-config\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.056439 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.056419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-trusted-ca-bundle\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.057734 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.057706    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-console-serving-cert\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.058073 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.058052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-console-oauth-config\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.063618 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.063600    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnbrd\" (UniqueName: \"kubernetes.io/projected/1c9a21e8-23c7-4e4c-92ab-bb91f419dbea-kube-api-access-lnbrd\") pod \"console-5c78ccbc66-dlwks\" (UID: \"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea\") " pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.160863 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.160823    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:22:58.442410 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.442387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c78ccbc66-dlwks"]
Sep 09 20:22:58.445306 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:22:58.445261    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9a21e8_23c7_4e4c_92ab_bb91f419dbea.slice/crio-7d0c4ff7e6de9bacc7b41f96981b9ba3d1ec5b6679cc8099925de0df3311b64e WatchSource:0}: Error finding container 7d0c4ff7e6de9bacc7b41f96981b9ba3d1ec5b6679cc8099925de0df3311b64e: Status 404 returned error can't find the container with id 7d0c4ff7e6de9bacc7b41f96981b9ba3d1ec5b6679cc8099925de0df3311b64e
Sep 09 20:22:58.770486 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.770373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c78ccbc66-dlwks" event={"ID":"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea","Type":"ContainerStarted","Data":"b2519984b83155a7a4a15ee73be82660eb983b79dea2b326b094c54716844ff2"}
Sep 09 20:22:58.770486 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.770447    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c78ccbc66-dlwks" event={"ID":"1c9a21e8-23c7-4e4c-92ab-bb91f419dbea","Type":"ContainerStarted","Data":"7d0c4ff7e6de9bacc7b41f96981b9ba3d1ec5b6679cc8099925de0df3311b64e"}
Sep 09 20:22:58.790526 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:22:58.790474    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c78ccbc66-dlwks" podStartSLOduration=1.7904571759999999 podStartE2EDuration="1.790457176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:58.788723823 +0000 UTC m=+400.281934820" watchObservedRunningTime="2026-09-09 20:22:58.790457176 +0000 UTC m=+400.283668160"
Sep 09 20:23:05.209791 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.209754    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"]
Sep 09 20:23:05.215960 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.215935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.218754 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.218733    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-28cn9\""
Sep 09 20:23:05.219859 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.219842    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 20:23:05.219958 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.219922    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 20:23:05.220029 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.219975    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 20:23:05.226901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.226877    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"]
Sep 09 20:23:05.315870 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.315837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6rv\" (UniqueName: \"kubernetes.io/projected/dc84bc43-83d8-4efc-a986-f1835827a75c-kube-api-access-sn6rv\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.316051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.315903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc84bc43-83d8-4efc-a986-f1835827a75c-metrics-cert\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.316051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.315922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dc84bc43-83d8-4efc-a986-f1835827a75c-manager-config\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.316051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.315981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc84bc43-83d8-4efc-a986-f1835827a75c-cert\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.417216 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.417187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc84bc43-83d8-4efc-a986-f1835827a75c-cert\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.417390 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.417223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6rv\" (UniqueName: \"kubernetes.io/projected/dc84bc43-83d8-4efc-a986-f1835827a75c-kube-api-access-sn6rv\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.417390 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.417277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc84bc43-83d8-4efc-a986-f1835827a75c-metrics-cert\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.417390 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.417307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dc84bc43-83d8-4efc-a986-f1835827a75c-manager-config\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.418095 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.418006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dc84bc43-83d8-4efc-a986-f1835827a75c-manager-config\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.419924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.419901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc84bc43-83d8-4efc-a986-f1835827a75c-metrics-cert\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.420015 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.419957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc84bc43-83d8-4efc-a986-f1835827a75c-cert\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.427162 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.427138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6rv\" (UniqueName: \"kubernetes.io/projected/dc84bc43-83d8-4efc-a986-f1835827a75c-kube-api-access-sn6rv\") pod \"lws-controller-manager-5bc9df88-4gffz\" (UID: \"dc84bc43-83d8-4efc-a986-f1835827a75c\") " pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.526187 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.526103    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:05.803753 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:05.803728    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"]
Sep 09 20:23:05.806069 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:23:05.806038    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc84bc43_83d8_4efc_a986_f1835827a75c.slice/crio-2e90146a1e3b9ec9b26210e57edeedeeebf5b0cd33046dd684492f0183016a28 WatchSource:0}: Error finding container 2e90146a1e3b9ec9b26210e57edeedeeebf5b0cd33046dd684492f0183016a28: Status 404 returned error can't find the container with id 2e90146a1e3b9ec9b26210e57edeedeeebf5b0cd33046dd684492f0183016a28
Sep 09 20:23:06.814170 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:06.814124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz" event={"ID":"dc84bc43-83d8-4efc-a986-f1835827a75c","Type":"ContainerStarted","Data":"2e90146a1e3b9ec9b26210e57edeedeeebf5b0cd33046dd684492f0183016a28"}
Sep 09 20:23:07.821034 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:07.821000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz" event={"ID":"dc84bc43-83d8-4efc-a986-f1835827a75c","Type":"ContainerStarted","Data":"2379855b93bc14642bde134cbd7d95fa5536c248d99d99b74e4b8d4f8a1520ff"}
Sep 09 20:23:07.821496 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:07.821068    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:07.845615 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:07.845546    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz" podStartSLOduration=1.238452315 podStartE2EDuration="2.845522604s" totalImagesPullingTime="1.607070289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:07.844005857 +0000 UTC m=+409.337216842" watchObservedRunningTime="2026-09-09 20:23:07.845522604 +0000 UTC m=+409.338733589"
Sep 09 20:23:08.161312 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:08.161226    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:23:08.161312 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:08.161288    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:23:08.166911 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:08.166884    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:23:08.832102 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:08.832064    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c78ccbc66-dlwks"
Sep 09 20:23:08.907919 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:08.907883    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8567f7f74-bhxx7"]
Sep 09 20:23:10.000217 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.000181    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"]
Sep 09 20:23:10.000666 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.000395    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d" podUID="b05d7159-1654-4248-97c7-6af9ca22a523" containerName="discovery" containerID="cri-o://70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934" gracePeriod=30
Sep 09 20:23:10.249143 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.249111    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:23:10.362456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.362369    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plf7b\" (UniqueName: \"kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-kube-api-access-plf7b\") pod \"b05d7159-1654-4248-97c7-6af9ca22a523\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") "
Sep 09 20:23:10.362456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.362409    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-cacerts\") pod \"b05d7159-1654-4248-97c7-6af9ca22a523\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") "
Sep 09 20:23:10.362456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.362431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05d7159-1654-4248-97c7-6af9ca22a523-local-certs\") pod \"b05d7159-1654-4248-97c7-6af9ca22a523\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") "
Sep 09 20:23:10.362456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.362453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-dns-cert\") pod \"b05d7159-1654-4248-97c7-6af9ca22a523\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") "
Sep 09 20:23:10.362785 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.362470    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-kubeconfig\") pod \"b05d7159-1654-4248-97c7-6af9ca22a523\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") "
Sep 09 20:23:10.362785 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.362486    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-ca-configmap\") pod \"b05d7159-1654-4248-97c7-6af9ca22a523\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") "
Sep 09 20:23:10.362785 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.362503    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-istio-token\") pod \"b05d7159-1654-4248-97c7-6af9ca22a523\" (UID: \"b05d7159-1654-4248-97c7-6af9ca22a523\") "
Sep 09 20:23:10.362968 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.362935    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b05d7159-1654-4248-97c7-6af9ca22a523" (UID: "b05d7159-1654-4248-97c7-6af9ca22a523"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:10.364957 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.364926    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b05d7159-1654-4248-97c7-6af9ca22a523" (UID: "b05d7159-1654-4248-97c7-6af9ca22a523"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:10.365106 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.364966    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b05d7159-1654-4248-97c7-6af9ca22a523" (UID: "b05d7159-1654-4248-97c7-6af9ca22a523"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:10.365258 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.365233    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-kube-api-access-plf7b" (OuterVolumeSpecName: "kube-api-access-plf7b") pod "b05d7159-1654-4248-97c7-6af9ca22a523" (UID: "b05d7159-1654-4248-97c7-6af9ca22a523"). InnerVolumeSpecName "kube-api-access-plf7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:23:10.365486 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.365453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-cacerts" (OuterVolumeSpecName: "cacerts") pod "b05d7159-1654-4248-97c7-6af9ca22a523" (UID: "b05d7159-1654-4248-97c7-6af9ca22a523"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:10.365575 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.365519    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05d7159-1654-4248-97c7-6af9ca22a523-local-certs" (OuterVolumeSpecName: "local-certs") pod "b05d7159-1654-4248-97c7-6af9ca22a523" (UID: "b05d7159-1654-4248-97c7-6af9ca22a523"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:23:10.365575 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.365520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-istio-token" (OuterVolumeSpecName: "istio-token") pod "b05d7159-1654-4248-97c7-6af9ca22a523" (UID: "b05d7159-1654-4248-97c7-6af9ca22a523"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:23:10.463239 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.463208    2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-cacerts\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:10.463239 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.463234    2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05d7159-1654-4248-97c7-6af9ca22a523-local-certs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:10.463239 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.463245    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-dns-cert\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:10.463502 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.463255    2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b05d7159-1654-4248-97c7-6af9ca22a523-istio-kubeconfig\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:10.463502 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.463264    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b05d7159-1654-4248-97c7-6af9ca22a523-istio-csr-ca-configmap\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:10.463502 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.463271    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-istio-token\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:10.463502 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.463279    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plf7b\" (UniqueName: \"kubernetes.io/projected/b05d7159-1654-4248-97c7-6af9ca22a523-kube-api-access-plf7b\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:10.837122 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.837087    2581 generic.go:356] "Generic (PLEG): container finished" podID="b05d7159-1654-4248-97c7-6af9ca22a523" containerID="70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934" exitCode=0
Sep 09 20:23:10.837329 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.837155    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"
Sep 09 20:23:10.837329 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.837179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d" event={"ID":"b05d7159-1654-4248-97c7-6af9ca22a523","Type":"ContainerDied","Data":"70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934"}
Sep 09 20:23:10.837329 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.837238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d" event={"ID":"b05d7159-1654-4248-97c7-6af9ca22a523","Type":"ContainerDied","Data":"a796bf22541b3e0f8a03ea4182e7b38dcbea071a189ccaf7f32ff165835f2aba"}
Sep 09 20:23:10.837329 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.837261    2581 scope.go:117] "RemoveContainer" containerID="70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934"
Sep 09 20:23:10.847092 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.847072    2581 scope.go:117] "RemoveContainer" containerID="70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934"
Sep 09 20:23:10.847442 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:23:10.847414    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934\": container with ID starting with 70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934 not found: ID does not exist" containerID="70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934"
Sep 09 20:23:10.847522 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.847456    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934"} err="failed to get container status \"70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934\": rpc error: code = NotFound desc = could not find container \"70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934\": container with ID starting with 70ef47e043755e188b1b570ba93742441d9c55521b9d412fb534feda7bcd4934 not found: ID does not exist"
Sep 09 20:23:10.859312 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.859286    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"]
Sep 09 20:23:10.863483 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:10.863461    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-rfx7d"]
Sep 09 20:23:11.001722 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:11.001693    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b05d7159-1654-4248-97c7-6af9ca22a523" path="/var/lib/kubelet/pods/b05d7159-1654-4248-97c7-6af9ca22a523/volumes"
Sep 09 20:23:18.829276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:18.829243    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5bc9df88-4gffz"
Sep 09 20:23:33.937484 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:33.937370    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8567f7f74-bhxx7" podUID="c8f6a62f-4e94-47e7-9dd4-eff95079e538" containerName="console" containerID="cri-o://f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d" gracePeriod=15
Sep 09 20:23:34.181031 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.180997    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:23:34.261104 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.261026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55j4j\" (UniqueName: \"kubernetes.io/projected/c8f6a62f-4e94-47e7-9dd4-eff95079e538-kube-api-access-55j4j\") pod \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") "
Sep 09 20:23:34.261104 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.261071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-oauth-serving-cert\") pod \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") "
Sep 09 20:23:34.261327 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.261142    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-trusted-ca-bundle\") pod \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") "
Sep 09 20:23:34.261327 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.261174    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-oauth-config\") pod \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") "
Sep 09 20:23:34.261327 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.261195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-service-ca\") pod \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") "
Sep 09 20:23:34.261327 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.261222    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-config\") pod \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") "
Sep 09 20:23:34.261327 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.261244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-serving-cert\") pod \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\" (UID: \"c8f6a62f-4e94-47e7-9dd4-eff95079e538\") "
Sep 09 20:23:34.261601 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.261580    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c8f6a62f-4e94-47e7-9dd4-eff95079e538" (UID: "c8f6a62f-4e94-47e7-9dd4-eff95079e538"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:34.261665 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.261600    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c8f6a62f-4e94-47e7-9dd4-eff95079e538" (UID: "c8f6a62f-4e94-47e7-9dd4-eff95079e538"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:34.261723 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.261700    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-service-ca" (OuterVolumeSpecName: "service-ca") pod "c8f6a62f-4e94-47e7-9dd4-eff95079e538" (UID: "c8f6a62f-4e94-47e7-9dd4-eff95079e538"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:34.261723 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.261697    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-config" (OuterVolumeSpecName: "console-config") pod "c8f6a62f-4e94-47e7-9dd4-eff95079e538" (UID: "c8f6a62f-4e94-47e7-9dd4-eff95079e538"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:23:34.263514 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.263491    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c8f6a62f-4e94-47e7-9dd4-eff95079e538" (UID: "c8f6a62f-4e94-47e7-9dd4-eff95079e538"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:34.263514 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.263505    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f6a62f-4e94-47e7-9dd4-eff95079e538-kube-api-access-55j4j" (OuterVolumeSpecName: "kube-api-access-55j4j") pod "c8f6a62f-4e94-47e7-9dd4-eff95079e538" (UID: "c8f6a62f-4e94-47e7-9dd4-eff95079e538"). InnerVolumeSpecName "kube-api-access-55j4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:23:34.263698 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.263681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c8f6a62f-4e94-47e7-9dd4-eff95079e538" (UID: "c8f6a62f-4e94-47e7-9dd4-eff95079e538"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:23:34.362392 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.362364    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-trusted-ca-bundle\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:34.362392 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.362390    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-oauth-config\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:34.362549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.362400    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:34.362549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.362408    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-config\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:34.362549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.362415    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8f6a62f-4e94-47e7-9dd4-eff95079e538-console-serving-cert\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:34.362549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.362424    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55j4j\" (UniqueName: \"kubernetes.io/projected/c8f6a62f-4e94-47e7-9dd4-eff95079e538-kube-api-access-55j4j\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:34.362549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.362432    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8f6a62f-4e94-47e7-9dd4-eff95079e538-oauth-serving-cert\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:23:34.954241 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.954205    2581 generic.go:356] "Generic (PLEG): container finished" podID="c8f6a62f-4e94-47e7-9dd4-eff95079e538" containerID="f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d" exitCode=0
Sep 09 20:23:34.954675 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.954254    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8567f7f74-bhxx7"
Sep 09 20:23:34.954675 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.954301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8567f7f74-bhxx7" event={"ID":"c8f6a62f-4e94-47e7-9dd4-eff95079e538","Type":"ContainerDied","Data":"f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d"}
Sep 09 20:23:34.954675 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.954400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8567f7f74-bhxx7" event={"ID":"c8f6a62f-4e94-47e7-9dd4-eff95079e538","Type":"ContainerDied","Data":"dc2f1e9c0dac58929c8b7067e6fcff57b3de03b9005c4e0ffcc3d39a7739512d"}
Sep 09 20:23:34.954675 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.954426    2581 scope.go:117] "RemoveContainer" containerID="f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d"
Sep 09 20:23:34.963774 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.963757    2581 scope.go:117] "RemoveContainer" containerID="f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d"
Sep 09 20:23:34.964047 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:23:34.964026    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d\": container with ID starting with f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d not found: ID does not exist" containerID="f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d"
Sep 09 20:23:34.964104 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.964056    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d"} err="failed to get container status \"f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d\": rpc error: code = NotFound desc = could not find container \"f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d\": container with ID starting with f121a305bcb1258154403e923bc91d1ddc46dd4678fd5c21f52665d8d61bdd4d not found: ID does not exist"
Sep 09 20:23:34.975832 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.975808    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8567f7f74-bhxx7"]
Sep 09 20:23:34.979368 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:34.979330    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8567f7f74-bhxx7"]
Sep 09 20:23:35.001511 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:23:35.001491    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8f6a62f-4e94-47e7-9dd4-eff95079e538" path="/var/lib/kubelet/pods/c8f6a62f-4e94-47e7-9dd4-eff95079e538/volumes"
Sep 09 20:24:17.022437 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.022404    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"]
Sep 09 20:24:17.022999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.022848    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f6a62f-4e94-47e7-9dd4-eff95079e538" containerName="console"
Sep 09 20:24:17.022999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.022866    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f6a62f-4e94-47e7-9dd4-eff95079e538" containerName="console"
Sep 09 20:24:17.022999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.022879    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b05d7159-1654-4248-97c7-6af9ca22a523" containerName="discovery"
Sep 09 20:24:17.022999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.022887    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b05d7159-1654-4248-97c7-6af9ca22a523" containerName="discovery"
Sep 09 20:24:17.022999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.022963    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b05d7159-1654-4248-97c7-6af9ca22a523" containerName="discovery"
Sep 09 20:24:17.022999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.022978    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f6a62f-4e94-47e7-9dd4-eff95079e538" containerName="console"
Sep 09 20:24:17.026026 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.026004    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:17.028788 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.028769    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:24:17.029331 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.029310    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mwfck\""
Sep 09 20:24:17.030006 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.029990    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:24:17.046092 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.046068    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"]
Sep 09 20:24:17.098215 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.098190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:17.098330 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.098219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9z7w\" (UniqueName: \"kubernetes.io/projected/46c30e7b-d6d3-4d68-adec-c83f503140c4-kube-api-access-h9z7w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:17.098330 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.098244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:17.131793 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.131767    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"]
Sep 09 20:24:17.135082 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.135067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:17.151334 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.151308    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"]
Sep 09 20:24:17.190281 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.190255    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"]
Sep 09 20:24:17.193548 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.193531    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:17.198786 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.198764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4j6\" (UniqueName: \"kubernetes.io/projected/b0191c0c-7537-4b35-b7d1-ddab59541bb7-kube-api-access-vz4j6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:17.198901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.198827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:17.198901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.198848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9z7w\" (UniqueName: \"kubernetes.io/projected/46c30e7b-d6d3-4d68-adec-c83f503140c4-kube-api-access-h9z7w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:17.198901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.198874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:17.198901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.198889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:17.199112 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.198927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:17.199188 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.199170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:17.199244 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.199206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:17.203494 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.203461    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"]
Sep 09 20:24:17.208600 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.208578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9z7w\" (UniqueName: \"kubernetes.io/projected/46c30e7b-d6d3-4d68-adec-c83f503140c4-kube-api-access-h9z7w\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:17.285196 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.285108    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"]
Sep 09 20:24:17.288610 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.288593    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:17.299322 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.299299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:17.299480 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.299381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:17.299480 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.299453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4j6\" (UniqueName: \"kubernetes.io/projected/b0191c0c-7537-4b35-b7d1-ddab59541bb7-kube-api-access-vz4j6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:17.299601 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.299550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:17.299660 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.299632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:17.299716 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.299676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:17.299812 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.299662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbhdv\" (UniqueName: \"kubernetes.io/projected/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-kube-api-access-cbhdv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:17.299959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.299942    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:17.301137 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.301110    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"]
Sep 09 20:24:17.307942 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.307918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4j6\" (UniqueName: \"kubernetes.io/projected/b0191c0c-7537-4b35-b7d1-ddab59541bb7-kube-api-access-vz4j6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:17.335152 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.335128    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:17.401419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.400792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:17.401419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.400849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbhdv\" (UniqueName: \"kubernetes.io/projected/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-kube-api-access-cbhdv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:17.401419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.400902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:17.401419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.400935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:17.401419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.400991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d8zn\" (UniqueName: \"kubernetes.io/projected/a799a7fa-23bb-41b8-9688-9977183b91a2-kube-api-access-6d8zn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:17.401419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.401035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:17.401419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.401092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:17.401419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.401370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:17.412736 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.412705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbhdv\" (UniqueName: \"kubernetes.io/projected/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-kube-api-access-cbhdv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:17.445422 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.445399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:17.502694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.502164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:17.502694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.502252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8zn\" (UniqueName: \"kubernetes.io/projected/a799a7fa-23bb-41b8-9688-9977183b91a2-kube-api-access-6d8zn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:17.502694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.502297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:17.502922 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.502794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:17.503399 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.503046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:17.503399 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.503154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:17.512439 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.512406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d8zn\" (UniqueName: \"kubernetes.io/projected/a799a7fa-23bb-41b8-9688-9977183b91a2-kube-api-access-6d8zn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:17.597932 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.597902    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:17.607941 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.607917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"]
Sep 09 20:24:17.610326 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:24:17.610304    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c30e7b_d6d3_4d68_adec_c83f503140c4.slice/crio-bcad374ade8118e71e931fd865b0953dbeddbcf8cc97bf5a0783e109737436c8 WatchSource:0}: Error finding container bcad374ade8118e71e931fd865b0953dbeddbcf8cc97bf5a0783e109737436c8: Status 404 returned error can't find the container with id bcad374ade8118e71e931fd865b0953dbeddbcf8cc97bf5a0783e109737436c8
Sep 09 20:24:17.722021 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.721998    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"]
Sep 09 20:24:17.723913 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:24:17.723878    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0191c0c_7537_4b35_b7d1_ddab59541bb7.slice/crio-7bed729955f4db9fa463efb01895dd0a9ca6c59002f6ead79cec79eb36629688 WatchSource:0}: Error finding container 7bed729955f4db9fa463efb01895dd0a9ca6c59002f6ead79cec79eb36629688: Status 404 returned error can't find the container with id 7bed729955f4db9fa463efb01895dd0a9ca6c59002f6ead79cec79eb36629688
Sep 09 20:24:17.790305 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.790274    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"]
Sep 09 20:24:17.791784 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:24:17.791763    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bbb7fb_15a9_45d6_aa23_1e6dd0cddfb0.slice/crio-2ea59a951efb086851e3090c1b9bd42be4ea67083873fc1f23c09704dcdc2e94 WatchSource:0}: Error finding container 2ea59a951efb086851e3090c1b9bd42be4ea67083873fc1f23c09704dcdc2e94: Status 404 returned error can't find the container with id 2ea59a951efb086851e3090c1b9bd42be4ea67083873fc1f23c09704dcdc2e94
Sep 09 20:24:17.876810 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:17.876714    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"]
Sep 09 20:24:17.942167 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:24:17.942134    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda799a7fa_23bb_41b8_9688_9977183b91a2.slice/crio-54ab4975b10ef60f766a91073cb7185a310bc7d6629d1fdd94009fc969fcc7dc WatchSource:0}: Error finding container 54ab4975b10ef60f766a91073cb7185a310bc7d6629d1fdd94009fc969fcc7dc: Status 404 returned error can't find the container with id 54ab4975b10ef60f766a91073cb7185a310bc7d6629d1fdd94009fc969fcc7dc
Sep 09 20:24:18.152315 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.152235    2581 generic.go:356] "Generic (PLEG): container finished" podID="46c30e7b-d6d3-4d68-adec-c83f503140c4" containerID="0402ac166d64a94e6ea43f14714d3ade6077a75459bdb5da37f36c30ac626295" exitCode=0
Sep 09 20:24:18.152724 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.152320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf" event={"ID":"46c30e7b-d6d3-4d68-adec-c83f503140c4","Type":"ContainerDied","Data":"0402ac166d64a94e6ea43f14714d3ade6077a75459bdb5da37f36c30ac626295"}
Sep 09 20:24:18.152724 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.152413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf" event={"ID":"46c30e7b-d6d3-4d68-adec-c83f503140c4","Type":"ContainerStarted","Data":"bcad374ade8118e71e931fd865b0953dbeddbcf8cc97bf5a0783e109737436c8"}
Sep 09 20:24:18.154128 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.154102    2581 generic.go:356] "Generic (PLEG): container finished" podID="94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" containerID="05b3a892fc3842f439ef69332262e80cad4dbe380ad770c24a7e3486a06e92d0" exitCode=0
Sep 09 20:24:18.154235 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.154173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq" event={"ID":"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0","Type":"ContainerDied","Data":"05b3a892fc3842f439ef69332262e80cad4dbe380ad770c24a7e3486a06e92d0"}
Sep 09 20:24:18.154235 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.154205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq" event={"ID":"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0","Type":"ContainerStarted","Data":"2ea59a951efb086851e3090c1b9bd42be4ea67083873fc1f23c09704dcdc2e94"}
Sep 09 20:24:18.155922 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.155903    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0191c0c-7537-4b35-b7d1-ddab59541bb7" containerID="51b8cc5e5d1e7222b085902d307aef4c4caf8f1b7d4c2f8ffe2e7c5dc4a989eb" exitCode=0
Sep 09 20:24:18.156035 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.155977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk" event={"ID":"b0191c0c-7537-4b35-b7d1-ddab59541bb7","Type":"ContainerDied","Data":"51b8cc5e5d1e7222b085902d307aef4c4caf8f1b7d4c2f8ffe2e7c5dc4a989eb"}
Sep 09 20:24:18.156035 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.156023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk" event={"ID":"b0191c0c-7537-4b35-b7d1-ddab59541bb7","Type":"ContainerStarted","Data":"7bed729955f4db9fa463efb01895dd0a9ca6c59002f6ead79cec79eb36629688"}
Sep 09 20:24:18.157760 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.157736    2581 generic.go:356] "Generic (PLEG): container finished" podID="a799a7fa-23bb-41b8-9688-9977183b91a2" containerID="e4312e2ef8aa3bc5c2b568fd0af63cf7b292c358f40fbfec9b0ac39f18002a85" exitCode=0
Sep 09 20:24:18.157880 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.157779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88" event={"ID":"a799a7fa-23bb-41b8-9688-9977183b91a2","Type":"ContainerDied","Data":"e4312e2ef8aa3bc5c2b568fd0af63cf7b292c358f40fbfec9b0ac39f18002a85"}
Sep 09 20:24:18.157880 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:18.157827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88" event={"ID":"a799a7fa-23bb-41b8-9688-9977183b91a2","Type":"ContainerStarted","Data":"54ab4975b10ef60f766a91073cb7185a310bc7d6629d1fdd94009fc969fcc7dc"}
Sep 09 20:24:19.164959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:19.164921    2581 generic.go:356] "Generic (PLEG): container finished" podID="46c30e7b-d6d3-4d68-adec-c83f503140c4" containerID="e5684b73ee87bed301054301132472da1045a907abfdb8fda43da301e2602c36" exitCode=0
Sep 09 20:24:19.165323 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:19.165004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf" event={"ID":"46c30e7b-d6d3-4d68-adec-c83f503140c4","Type":"ContainerDied","Data":"e5684b73ee87bed301054301132472da1045a907abfdb8fda43da301e2602c36"}
Sep 09 20:24:19.166899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:19.166876    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0191c0c-7537-4b35-b7d1-ddab59541bb7" containerID="b8193f191f93ac7b0971399df74501880809dd842a38d6765868cea4e29a450e" exitCode=0
Sep 09 20:24:19.167004 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:19.166934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk" event={"ID":"b0191c0c-7537-4b35-b7d1-ddab59541bb7","Type":"ContainerDied","Data":"b8193f191f93ac7b0971399df74501880809dd842a38d6765868cea4e29a450e"}
Sep 09 20:24:20.174572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:20.174533    2581 generic.go:356] "Generic (PLEG): container finished" podID="46c30e7b-d6d3-4d68-adec-c83f503140c4" containerID="ae29d4ff72698f03ec621a7ec0f4f9cda0666e9b1e8faa61be931584f4d21082" exitCode=0
Sep 09 20:24:20.174980 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:20.174622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf" event={"ID":"46c30e7b-d6d3-4d68-adec-c83f503140c4","Type":"ContainerDied","Data":"ae29d4ff72698f03ec621a7ec0f4f9cda0666e9b1e8faa61be931584f4d21082"}
Sep 09 20:24:20.176467 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:20.176445    2581 generic.go:356] "Generic (PLEG): container finished" podID="94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" containerID="66987ded0d888467e640fa84ee5641e27a74c42cccb09bd17f0b994147e1dc21" exitCode=0
Sep 09 20:24:20.176588 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:20.176516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq" event={"ID":"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0","Type":"ContainerDied","Data":"66987ded0d888467e640fa84ee5641e27a74c42cccb09bd17f0b994147e1dc21"}
Sep 09 20:24:20.178609 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:20.178592    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0191c0c-7537-4b35-b7d1-ddab59541bb7" containerID="c771f9b6286e67a30a41e63e06bdc109c3ec8e41418f8b80842b572a683c631f" exitCode=0
Sep 09 20:24:20.178701 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:20.178678    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk" event={"ID":"b0191c0c-7537-4b35-b7d1-ddab59541bb7","Type":"ContainerDied","Data":"c771f9b6286e67a30a41e63e06bdc109c3ec8e41418f8b80842b572a683c631f"}
Sep 09 20:24:20.180304 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:20.180287    2581 generic.go:356] "Generic (PLEG): container finished" podID="a799a7fa-23bb-41b8-9688-9977183b91a2" containerID="9885dff268d732bf21afbf4e29c6b0bb6ec55da1a79d010d782da10c8c2a7113" exitCode=0
Sep 09 20:24:20.180414 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:20.180327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88" event={"ID":"a799a7fa-23bb-41b8-9688-9977183b91a2","Type":"ContainerDied","Data":"9885dff268d732bf21afbf4e29c6b0bb6ec55da1a79d010d782da10c8c2a7113"}
Sep 09 20:24:21.187795 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.187758    2581 generic.go:356] "Generic (PLEG): container finished" podID="94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" containerID="ebd90b8716bac72d35b0a604577057e7ddfd49c0aa33e82209de07d360fe10fa" exitCode=0
Sep 09 20:24:21.188238 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.187838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq" event={"ID":"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0","Type":"ContainerDied","Data":"ebd90b8716bac72d35b0a604577057e7ddfd49c0aa33e82209de07d360fe10fa"}
Sep 09 20:24:21.190228 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.190199    2581 generic.go:356] "Generic (PLEG): container finished" podID="a799a7fa-23bb-41b8-9688-9977183b91a2" containerID="21ffaee38e90a81d02e6932710c5643fe1152b7e8a0523c9c788006892e3f4dc" exitCode=0
Sep 09 20:24:21.190389 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.190268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88" event={"ID":"a799a7fa-23bb-41b8-9688-9977183b91a2","Type":"ContainerDied","Data":"21ffaee38e90a81d02e6932710c5643fe1152b7e8a0523c9c788006892e3f4dc"}
Sep 09 20:24:21.345178 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.345155    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:21.348286 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.348265    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:21.438327 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.438257    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-bundle\") pod \"46c30e7b-d6d3-4d68-adec-c83f503140c4\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") "
Sep 09 20:24:21.438327 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.438285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-util\") pod \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") "
Sep 09 20:24:21.438327 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.438307    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-util\") pod \"46c30e7b-d6d3-4d68-adec-c83f503140c4\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") "
Sep 09 20:24:21.438327 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.438327    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-bundle\") pod \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") "
Sep 09 20:24:21.438666 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.438386    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz4j6\" (UniqueName: \"kubernetes.io/projected/b0191c0c-7537-4b35-b7d1-ddab59541bb7-kube-api-access-vz4j6\") pod \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\" (UID: \"b0191c0c-7537-4b35-b7d1-ddab59541bb7\") "
Sep 09 20:24:21.438666 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.438442    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9z7w\" (UniqueName: \"kubernetes.io/projected/46c30e7b-d6d3-4d68-adec-c83f503140c4-kube-api-access-h9z7w\") pod \"46c30e7b-d6d3-4d68-adec-c83f503140c4\" (UID: \"46c30e7b-d6d3-4d68-adec-c83f503140c4\") "
Sep 09 20:24:21.438867 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.438815    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-bundle" (OuterVolumeSpecName: "bundle") pod "b0191c0c-7537-4b35-b7d1-ddab59541bb7" (UID: "b0191c0c-7537-4b35-b7d1-ddab59541bb7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:21.439102 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.439046    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-bundle" (OuterVolumeSpecName: "bundle") pod "46c30e7b-d6d3-4d68-adec-c83f503140c4" (UID: "46c30e7b-d6d3-4d68-adec-c83f503140c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:21.441146 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.441101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c30e7b-d6d3-4d68-adec-c83f503140c4-kube-api-access-h9z7w" (OuterVolumeSpecName: "kube-api-access-h9z7w") pod "46c30e7b-d6d3-4d68-adec-c83f503140c4" (UID: "46c30e7b-d6d3-4d68-adec-c83f503140c4"). InnerVolumeSpecName "kube-api-access-h9z7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:21.441222 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.441146    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0191c0c-7537-4b35-b7d1-ddab59541bb7-kube-api-access-vz4j6" (OuterVolumeSpecName: "kube-api-access-vz4j6") pod "b0191c0c-7537-4b35-b7d1-ddab59541bb7" (UID: "b0191c0c-7537-4b35-b7d1-ddab59541bb7"). InnerVolumeSpecName "kube-api-access-vz4j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:21.444648 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.444628    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-util" (OuterVolumeSpecName: "util") pod "46c30e7b-d6d3-4d68-adec-c83f503140c4" (UID: "46c30e7b-d6d3-4d68-adec-c83f503140c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:21.444927 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.444908    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-util" (OuterVolumeSpecName: "util") pod "b0191c0c-7537-4b35-b7d1-ddab59541bb7" (UID: "b0191c0c-7537-4b35-b7d1-ddab59541bb7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:21.539276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.539246    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9z7w\" (UniqueName: \"kubernetes.io/projected/46c30e7b-d6d3-4d68-adec-c83f503140c4-kube-api-access-h9z7w\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:21.539276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.539269    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-bundle\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:21.539276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.539280    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-util\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:21.539276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.539287    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46c30e7b-d6d3-4d68-adec-c83f503140c4-util\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:21.539535 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.539296    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b0191c0c-7537-4b35-b7d1-ddab59541bb7-bundle\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:21.539535 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:21.539304    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz4j6\" (UniqueName: \"kubernetes.io/projected/b0191c0c-7537-4b35-b7d1-ddab59541bb7-kube-api-access-vz4j6\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:22.196999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.196966    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk"
Sep 09 20:24:22.197607 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.196993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xhxgk" event={"ID":"b0191c0c-7537-4b35-b7d1-ddab59541bb7","Type":"ContainerDied","Data":"7bed729955f4db9fa463efb01895dd0a9ca6c59002f6ead79cec79eb36629688"}
Sep 09 20:24:22.197607 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.197043    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bed729955f4db9fa463efb01895dd0a9ca6c59002f6ead79cec79eb36629688"
Sep 09 20:24:22.199090 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.199072    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf"
Sep 09 20:24:22.199282 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.199083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6csmf" event={"ID":"46c30e7b-d6d3-4d68-adec-c83f503140c4","Type":"ContainerDied","Data":"bcad374ade8118e71e931fd865b0953dbeddbcf8cc97bf5a0783e109737436c8"}
Sep 09 20:24:22.199282 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.199125    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcad374ade8118e71e931fd865b0953dbeddbcf8cc97bf5a0783e109737436c8"
Sep 09 20:24:22.351278 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.351257    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:22.353357 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.353321    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:22.446777 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.446748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-bundle\") pod \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") "
Sep 09 20:24:22.446959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.446787    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-util\") pod \"a799a7fa-23bb-41b8-9688-9977183b91a2\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") "
Sep 09 20:24:22.446959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.446813    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d8zn\" (UniqueName: \"kubernetes.io/projected/a799a7fa-23bb-41b8-9688-9977183b91a2-kube-api-access-6d8zn\") pod \"a799a7fa-23bb-41b8-9688-9977183b91a2\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") "
Sep 09 20:24:22.446959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.446849    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-util\") pod \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") "
Sep 09 20:24:22.446959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.446895    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbhdv\" (UniqueName: \"kubernetes.io/projected/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-kube-api-access-cbhdv\") pod \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\" (UID: \"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0\") "
Sep 09 20:24:22.446959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.446945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-bundle\") pod \"a799a7fa-23bb-41b8-9688-9977183b91a2\" (UID: \"a799a7fa-23bb-41b8-9688-9977183b91a2\") "
Sep 09 20:24:22.447468 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.447399    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-bundle" (OuterVolumeSpecName: "bundle") pod "94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" (UID: "94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:22.447757 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.447637    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-bundle" (OuterVolumeSpecName: "bundle") pod "a799a7fa-23bb-41b8-9688-9977183b91a2" (UID: "a799a7fa-23bb-41b8-9688-9977183b91a2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:22.449133 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.449096    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a799a7fa-23bb-41b8-9688-9977183b91a2-kube-api-access-6d8zn" (OuterVolumeSpecName: "kube-api-access-6d8zn") pod "a799a7fa-23bb-41b8-9688-9977183b91a2" (UID: "a799a7fa-23bb-41b8-9688-9977183b91a2"). InnerVolumeSpecName "kube-api-access-6d8zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:22.449235 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.449210    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-kube-api-access-cbhdv" (OuterVolumeSpecName: "kube-api-access-cbhdv") pod "94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" (UID: "94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0"). InnerVolumeSpecName "kube-api-access-cbhdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:24:22.452852 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.452826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-util" (OuterVolumeSpecName: "util") pod "a799a7fa-23bb-41b8-9688-9977183b91a2" (UID: "a799a7fa-23bb-41b8-9688-9977183b91a2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:22.452852 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.452843    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-util" (OuterVolumeSpecName: "util") pod "94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" (UID: "94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:24:22.548463 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.548431    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbhdv\" (UniqueName: \"kubernetes.io/projected/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-kube-api-access-cbhdv\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:22.548463 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.548458    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-bundle\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:22.548463 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.548469    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-bundle\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:22.548668 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.548477    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a799a7fa-23bb-41b8-9688-9977183b91a2-util\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:22.548668 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.548484    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d8zn\" (UniqueName: \"kubernetes.io/projected/a799a7fa-23bb-41b8-9688-9977183b91a2-kube-api-access-6d8zn\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:22.548668 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:22.548493    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0-util\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:24:23.206942 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:23.206904    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq"
Sep 09 20:24:23.206942 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:23.206915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxnrhq" event={"ID":"94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0","Type":"ContainerDied","Data":"2ea59a951efb086851e3090c1b9bd42be4ea67083873fc1f23c09704dcdc2e94"}
Sep 09 20:24:23.207499 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:23.206961    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ea59a951efb086851e3090c1b9bd42be4ea67083873fc1f23c09704dcdc2e94"
Sep 09 20:24:23.209171 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:23.209097    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88"
Sep 09 20:24:23.209171 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:23.209113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79vc88" event={"ID":"a799a7fa-23bb-41b8-9688-9977183b91a2","Type":"ContainerDied","Data":"54ab4975b10ef60f766a91073cb7185a310bc7d6629d1fdd94009fc969fcc7dc"}
Sep 09 20:24:23.209401 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:23.209179    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54ab4975b10ef60f766a91073cb7185a310bc7d6629d1fdd94009fc969fcc7dc"
Sep 09 20:24:43.912961 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.912924    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"]
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913230    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0191c0c-7537-4b35-b7d1-ddab59541bb7" containerName="extract"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913242    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0191c0c-7537-4b35-b7d1-ddab59541bb7" containerName="extract"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913250    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a799a7fa-23bb-41b8-9688-9977183b91a2" containerName="util"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913256    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a799a7fa-23bb-41b8-9688-9977183b91a2" containerName="util"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913261    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0191c0c-7537-4b35-b7d1-ddab59541bb7" containerName="pull"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913266    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0191c0c-7537-4b35-b7d1-ddab59541bb7" containerName="pull"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913272    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a799a7fa-23bb-41b8-9688-9977183b91a2" containerName="extract"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913277    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a799a7fa-23bb-41b8-9688-9977183b91a2" containerName="extract"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913284    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46c30e7b-d6d3-4d68-adec-c83f503140c4" containerName="util"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913291    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46c30e7b-d6d3-4d68-adec-c83f503140c4" containerName="util"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913301    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46c30e7b-d6d3-4d68-adec-c83f503140c4" containerName="pull"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913306    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46c30e7b-d6d3-4d68-adec-c83f503140c4" containerName="pull"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913317    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0191c0c-7537-4b35-b7d1-ddab59541bb7" containerName="util"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913322    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0191c0c-7537-4b35-b7d1-ddab59541bb7" containerName="util"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913327    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a799a7fa-23bb-41b8-9688-9977183b91a2" containerName="pull"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913331    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a799a7fa-23bb-41b8-9688-9977183b91a2" containerName="pull"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913337    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" containerName="pull"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913367    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" containerName="pull"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913408    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" containerName="extract"
Sep 09 20:24:43.913408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913416    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" containerName="extract"
Sep 09 20:24:43.914123 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913427    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" containerName="util"
Sep 09 20:24:43.914123 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913435    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" containerName="util"
Sep 09 20:24:43.914123 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913442    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46c30e7b-d6d3-4d68-adec-c83f503140c4" containerName="extract"
Sep 09 20:24:43.914123 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913450    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46c30e7b-d6d3-4d68-adec-c83f503140c4" containerName="extract"
Sep 09 20:24:43.914123 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913523    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="94bbb7fb-15a9-45d6-aa23-1e6dd0cddfb0" containerName="extract"
Sep 09 20:24:43.914123 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913536    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0191c0c-7537-4b35-b7d1-ddab59541bb7" containerName="extract"
Sep 09 20:24:43.914123 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913547    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46c30e7b-d6d3-4d68-adec-c83f503140c4" containerName="extract"
Sep 09 20:24:43.914123 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.913558    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a799a7fa-23bb-41b8-9688-9977183b91a2" containerName="extract"
Sep 09 20:24:43.916399 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.916385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"
Sep 09 20:24:43.919149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.919124    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 20:24:43.919149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.919147    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 20:24:43.919335 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.919186    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 20:24:43.919335 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.919277    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xl2tw\""
Sep 09 20:24:43.919486 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.919457    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 20:24:43.927259 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:43.927240    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"]
Sep 09 20:24:44.032901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:44.032867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8aeef901-a7d1-49e2-bab8-0c578fb19e1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zmhwn\" (UID: \"8aeef901-a7d1-49e2-bab8-0c578fb19e1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"
Sep 09 20:24:44.033059 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:44.032919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8aeef901-a7d1-49e2-bab8-0c578fb19e1c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zmhwn\" (UID: \"8aeef901-a7d1-49e2-bab8-0c578fb19e1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"
Sep 09 20:24:44.033059 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:44.032990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkkvn\" (UniqueName: \"kubernetes.io/projected/8aeef901-a7d1-49e2-bab8-0c578fb19e1c-kube-api-access-jkkvn\") pod \"kuadrant-console-plugin-fbdd4f445-zmhwn\" (UID: \"8aeef901-a7d1-49e2-bab8-0c578fb19e1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"
Sep 09 20:24:44.134306 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:44.134276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8aeef901-a7d1-49e2-bab8-0c578fb19e1c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zmhwn\" (UID: \"8aeef901-a7d1-49e2-bab8-0c578fb19e1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"
Sep 09 20:24:44.134484 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:44.134322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkkvn\" (UniqueName: \"kubernetes.io/projected/8aeef901-a7d1-49e2-bab8-0c578fb19e1c-kube-api-access-jkkvn\") pod \"kuadrant-console-plugin-fbdd4f445-zmhwn\" (UID: \"8aeef901-a7d1-49e2-bab8-0c578fb19e1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"
Sep 09 20:24:44.134484 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:44.134409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8aeef901-a7d1-49e2-bab8-0c578fb19e1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zmhwn\" (UID: \"8aeef901-a7d1-49e2-bab8-0c578fb19e1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"
Sep 09 20:24:44.135029 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:44.135001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8aeef901-a7d1-49e2-bab8-0c578fb19e1c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zmhwn\" (UID: \"8aeef901-a7d1-49e2-bab8-0c578fb19e1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"
Sep 09 20:24:44.136941 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:44.136915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8aeef901-a7d1-49e2-bab8-0c578fb19e1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zmhwn\" (UID: \"8aeef901-a7d1-49e2-bab8-0c578fb19e1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"
Sep 09 20:24:44.142929 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:44.142905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkkvn\" (UniqueName: \"kubernetes.io/projected/8aeef901-a7d1-49e2-bab8-0c578fb19e1c-kube-api-access-jkkvn\") pod \"kuadrant-console-plugin-fbdd4f445-zmhwn\" (UID: \"8aeef901-a7d1-49e2-bab8-0c578fb19e1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"
Sep 09 20:24:44.225262 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:44.225228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"
Sep 09 20:24:44.508729 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:44.508708    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn"]
Sep 09 20:24:44.510622 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:24:44.510597    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8aeef901_a7d1_49e2_bab8_0c578fb19e1c.slice/crio-1847fae9999e1afe5ba8572b641104abb33bcabe83684a1fbfabe7ee545751bb WatchSource:0}: Error finding container 1847fae9999e1afe5ba8572b641104abb33bcabe83684a1fbfabe7ee545751bb: Status 404 returned error can't find the container with id 1847fae9999e1afe5ba8572b641104abb33bcabe83684a1fbfabe7ee545751bb
Sep 09 20:24:45.329364 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:45.329304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn" event={"ID":"8aeef901-a7d1-49e2-bab8-0c578fb19e1c","Type":"ContainerStarted","Data":"1847fae9999e1afe5ba8572b641104abb33bcabe83684a1fbfabe7ee545751bb"}
Sep 09 20:24:49.353277 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:49.353241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn" event={"ID":"8aeef901-a7d1-49e2-bab8-0c578fb19e1c","Type":"ContainerStarted","Data":"d86a1cdd60cfe731547686c50bea63767c3d8f327c848ed4b848a70b3bbd3f3f"}
Sep 09 20:24:49.372542 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:24:49.372489    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zmhwn" podStartSLOduration=1.725068789 podStartE2EDuration="6.372467843s" totalImagesPullingTime="4.647399054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:24:49.370069238 +0000 UTC m=+510.863280233" watchObservedRunningTime="2026-09-09 20:24:49.372467843 +0000 UTC m=+510.865678828"
Sep 09 20:25:38.868810 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:38.868729    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-w8gth"]
Sep 09 20:25:38.883296 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:38.883267    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-w8gth"]
Sep 09 20:25:38.883464 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:38.883382    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth"
Sep 09 20:25:38.886169 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:38.886149    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9w79l\""
Sep 09 20:25:38.987527 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:38.987490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldljr\" (UniqueName: \"kubernetes.io/projected/2213590b-7906-491c-945f-495e98b5745e-kube-api-access-ldljr\") pod \"authorino-7bcfd8f9cd-w8gth\" (UID: \"2213590b-7906-491c-945f-495e98b5745e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth"
Sep 09 20:25:39.052940 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.052916    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-zxgsp"]
Sep 09 20:25:39.054998 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.054983    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zxgsp"
Sep 09 20:25:39.063318 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.063295    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zxgsp"]
Sep 09 20:25:39.088903 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.088877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldljr\" (UniqueName: \"kubernetes.io/projected/2213590b-7906-491c-945f-495e98b5745e-kube-api-access-ldljr\") pod \"authorino-7bcfd8f9cd-w8gth\" (UID: \"2213590b-7906-491c-945f-495e98b5745e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth"
Sep 09 20:25:39.098456 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.098431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldljr\" (UniqueName: \"kubernetes.io/projected/2213590b-7906-491c-945f-495e98b5745e-kube-api-access-ldljr\") pod \"authorino-7bcfd8f9cd-w8gth\" (UID: \"2213590b-7906-491c-945f-495e98b5745e\") " pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth"
Sep 09 20:25:39.189814 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.189782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvqn\" (UniqueName: \"kubernetes.io/projected/e3f99c15-9c26-4ac1-b525-d8a8242932f7-kube-api-access-nlvqn\") pod \"authorino-59c974bd48-zxgsp\" (UID: \"e3f99c15-9c26-4ac1-b525-d8a8242932f7\") " pod="kuadrant-system/authorino-59c974bd48-zxgsp"
Sep 09 20:25:39.192642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.192623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth"
Sep 09 20:25:39.291084 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.291046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvqn\" (UniqueName: \"kubernetes.io/projected/e3f99c15-9c26-4ac1-b525-d8a8242932f7-kube-api-access-nlvqn\") pod \"authorino-59c974bd48-zxgsp\" (UID: \"e3f99c15-9c26-4ac1-b525-d8a8242932f7\") " pod="kuadrant-system/authorino-59c974bd48-zxgsp"
Sep 09 20:25:39.299947 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.299912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvqn\" (UniqueName: \"kubernetes.io/projected/e3f99c15-9c26-4ac1-b525-d8a8242932f7-kube-api-access-nlvqn\") pod \"authorino-59c974bd48-zxgsp\" (UID: \"e3f99c15-9c26-4ac1-b525-d8a8242932f7\") " pod="kuadrant-system/authorino-59c974bd48-zxgsp"
Sep 09 20:25:39.363898 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.363872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zxgsp"
Sep 09 20:25:39.460909 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.460882    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-w8gth"]
Sep 09 20:25:39.463189 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:25:39.463163    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2213590b_7906_491c_945f_495e98b5745e.slice/crio-259588520bbadcca323c0c6869a217b1ea10939e254e8fad91736d2716cd02c4 WatchSource:0}: Error finding container 259588520bbadcca323c0c6869a217b1ea10939e254e8fad91736d2716cd02c4: Status 404 returned error can't find the container with id 259588520bbadcca323c0c6869a217b1ea10939e254e8fad91736d2716cd02c4
Sep 09 20:25:39.602001 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.601966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth" event={"ID":"2213590b-7906-491c-945f-495e98b5745e","Type":"ContainerStarted","Data":"259588520bbadcca323c0c6869a217b1ea10939e254e8fad91736d2716cd02c4"}
Sep 09 20:25:39.637831 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:39.637805    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zxgsp"]
Sep 09 20:25:39.640238 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:25:39.640213    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3f99c15_9c26_4ac1_b525_d8a8242932f7.slice/crio-145e22949e69cca3c4e04a88bac715ece9bd624204b15b734e827b2cd289297f WatchSource:0}: Error finding container 145e22949e69cca3c4e04a88bac715ece9bd624204b15b734e827b2cd289297f: Status 404 returned error can't find the container with id 145e22949e69cca3c4e04a88bac715ece9bd624204b15b734e827b2cd289297f
Sep 09 20:25:40.626363 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:40.622622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zxgsp" event={"ID":"e3f99c15-9c26-4ac1-b525-d8a8242932f7","Type":"ContainerStarted","Data":"145e22949e69cca3c4e04a88bac715ece9bd624204b15b734e827b2cd289297f"}
Sep 09 20:25:42.635040 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:42.634942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zxgsp" event={"ID":"e3f99c15-9c26-4ac1-b525-d8a8242932f7","Type":"ContainerStarted","Data":"aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b"}
Sep 09 20:25:42.636509 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:42.636485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth" event={"ID":"2213590b-7906-491c-945f-495e98b5745e","Type":"ContainerStarted","Data":"8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3"}
Sep 09 20:25:42.652380 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:42.652316    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-zxgsp" podStartSLOduration=0.943979923 podStartE2EDuration="3.652301583s" totalImagesPullingTime="2.70832166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:42.651645179 +0000 UTC m=+564.144856162" watchObservedRunningTime="2026-09-09 20:25:42.652301583 +0000 UTC m=+564.145512568"
Sep 09 20:25:42.678992 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:42.678944    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth" podStartSLOduration=1.783360013 podStartE2EDuration="4.678927853s" totalImagesPullingTime="2.89556784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:42.677593322 +0000 UTC m=+564.170804307" watchObservedRunningTime="2026-09-09 20:25:42.678927853 +0000 UTC m=+564.172138837"
Sep 09 20:25:42.708628 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:42.708598    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-w8gth"]
Sep 09 20:25:44.646062 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:44.645995    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth" podUID="2213590b-7906-491c-945f-495e98b5745e" containerName="authorino" containerID="cri-o://8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3" gracePeriod=30
Sep 09 20:25:44.892062 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:44.891995    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth"
Sep 09 20:25:45.037857 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.037830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldljr\" (UniqueName: \"kubernetes.io/projected/2213590b-7906-491c-945f-495e98b5745e-kube-api-access-ldljr\") pod \"2213590b-7906-491c-945f-495e98b5745e\" (UID: \"2213590b-7906-491c-945f-495e98b5745e\") "
Sep 09 20:25:45.040067 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.040042    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2213590b-7906-491c-945f-495e98b5745e-kube-api-access-ldljr" (OuterVolumeSpecName: "kube-api-access-ldljr") pod "2213590b-7906-491c-945f-495e98b5745e" (UID: "2213590b-7906-491c-945f-495e98b5745e"). InnerVolumeSpecName "kube-api-access-ldljr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:45.138749 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.138709    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldljr\" (UniqueName: \"kubernetes.io/projected/2213590b-7906-491c-945f-495e98b5745e-kube-api-access-ldljr\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:25:45.652174 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.652135    2581 generic.go:356] "Generic (PLEG): container finished" podID="2213590b-7906-491c-945f-495e98b5745e" containerID="8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3" exitCode=0
Sep 09 20:25:45.652174 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.652177    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth"
Sep 09 20:25:45.652694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.652230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth" event={"ID":"2213590b-7906-491c-945f-495e98b5745e","Type":"ContainerDied","Data":"8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3"}
Sep 09 20:25:45.652694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.652280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-w8gth" event={"ID":"2213590b-7906-491c-945f-495e98b5745e","Type":"ContainerDied","Data":"259588520bbadcca323c0c6869a217b1ea10939e254e8fad91736d2716cd02c4"}
Sep 09 20:25:45.652694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.652303    2581 scope.go:117] "RemoveContainer" containerID="8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3"
Sep 09 20:25:45.661166 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.661141    2581 scope.go:117] "RemoveContainer" containerID="8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3"
Sep 09 20:25:45.661493 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:25:45.661476    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3\": container with ID starting with 8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3 not found: ID does not exist" containerID="8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3"
Sep 09 20:25:45.661548 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.661501    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3"} err="failed to get container status \"8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3\": rpc error: code = NotFound desc = could not find container \"8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3\": container with ID starting with 8fce940ec2a61a64c6716172ebc0b08110aa198af5ef5b1a3a2e402055fad4a3 not found: ID does not exist"
Sep 09 20:25:45.675910 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.675877    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-w8gth"]
Sep 09 20:25:45.680503 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:45.680483    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-w8gth"]
Sep 09 20:25:47.001475 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:25:47.001442    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2213590b-7906-491c-945f-495e98b5745e" path="/var/lib/kubelet/pods/2213590b-7906-491c-945f-495e98b5745e/volumes"
Sep 09 20:26:08.680770 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.680736    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-dzfqg"]
Sep 09 20:26:08.681200 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.681111    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2213590b-7906-491c-945f-495e98b5745e" containerName="authorino"
Sep 09 20:26:08.681200 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.681124    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2213590b-7906-491c-945f-495e98b5745e" containerName="authorino"
Sep 09 20:26:08.681200 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.681179    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2213590b-7906-491c-945f-495e98b5745e" containerName="authorino"
Sep 09 20:26:08.683865 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.683849    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dzfqg"
Sep 09 20:26:08.688579 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.688552    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-dzfqg"]
Sep 09 20:26:08.710769 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.710742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jh52\" (UniqueName: \"kubernetes.io/projected/a4ec44b7-98b7-4276-90cb-14e2766fc1a3-kube-api-access-5jh52\") pod \"authorino-557f876f69-dzfqg\" (UID: \"a4ec44b7-98b7-4276-90cb-14e2766fc1a3\") " pod="kuadrant-system/authorino-557f876f69-dzfqg"
Sep 09 20:26:08.811529 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.811494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jh52\" (UniqueName: \"kubernetes.io/projected/a4ec44b7-98b7-4276-90cb-14e2766fc1a3-kube-api-access-5jh52\") pod \"authorino-557f876f69-dzfqg\" (UID: \"a4ec44b7-98b7-4276-90cb-14e2766fc1a3\") " pod="kuadrant-system/authorino-557f876f69-dzfqg"
Sep 09 20:26:08.819839 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.819814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jh52\" (UniqueName: \"kubernetes.io/projected/a4ec44b7-98b7-4276-90cb-14e2766fc1a3-kube-api-access-5jh52\") pod \"authorino-557f876f69-dzfqg\" (UID: \"a4ec44b7-98b7-4276-90cb-14e2766fc1a3\") " pod="kuadrant-system/authorino-557f876f69-dzfqg"
Sep 09 20:26:08.938265 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.938173    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dzfqg"]
Sep 09 20:26:08.938506 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.938466    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dzfqg"
Sep 09 20:26:08.966037 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.965994    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d58666bd-5cbnj"]
Sep 09 20:26:08.968315 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.968294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d58666bd-5cbnj"
Sep 09 20:26:08.978301 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:08.978275    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d58666bd-5cbnj"]
Sep 09 20:26:09.012945 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:09.012917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfgb\" (UniqueName: \"kubernetes.io/projected/3f53716a-dba1-452f-a392-66cec3b5d01e-kube-api-access-blfgb\") pod \"authorino-7d58666bd-5cbnj\" (UID: \"3f53716a-dba1-452f-a392-66cec3b5d01e\") " pod="kuadrant-system/authorino-7d58666bd-5cbnj"
Sep 09 20:26:09.113463 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:09.113436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfgb\" (UniqueName: \"kubernetes.io/projected/3f53716a-dba1-452f-a392-66cec3b5d01e-kube-api-access-blfgb\") pod \"authorino-7d58666bd-5cbnj\" (UID: \"3f53716a-dba1-452f-a392-66cec3b5d01e\") " pod="kuadrant-system/authorino-7d58666bd-5cbnj"
Sep 09 20:26:09.125364 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:09.125303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfgb\" (UniqueName: \"kubernetes.io/projected/3f53716a-dba1-452f-a392-66cec3b5d01e-kube-api-access-blfgb\") pod \"authorino-7d58666bd-5cbnj\" (UID: \"3f53716a-dba1-452f-a392-66cec3b5d01e\") " pod="kuadrant-system/authorino-7d58666bd-5cbnj"
Sep 09 20:26:09.208220 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:09.208191    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dzfqg"]
Sep 09 20:26:09.209500 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:26:09.209472    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4ec44b7_98b7_4276_90cb_14e2766fc1a3.slice/crio-8e61cf203ffe2c637ba7e97046aa717a6a58227c2bf8268449848a737b1f4d70 WatchSource:0}: Error finding container 8e61cf203ffe2c637ba7e97046aa717a6a58227c2bf8268449848a737b1f4d70: Status 404 returned error can't find the container with id 8e61cf203ffe2c637ba7e97046aa717a6a58227c2bf8268449848a737b1f4d70
Sep 09 20:26:09.279401 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:09.279330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d58666bd-5cbnj"
Sep 09 20:26:09.549529 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:09.549495    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d58666bd-5cbnj"]
Sep 09 20:26:09.550223 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:26:09.550199    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f53716a_dba1_452f_a392_66cec3b5d01e.slice/crio-0bde600f258f1fc5bcbbb57982118652cfa9d7d1bb2d03ad63da6aea17498425 WatchSource:0}: Error finding container 0bde600f258f1fc5bcbbb57982118652cfa9d7d1bb2d03ad63da6aea17498425: Status 404 returned error can't find the container with id 0bde600f258f1fc5bcbbb57982118652cfa9d7d1bb2d03ad63da6aea17498425
Sep 09 20:26:09.780769 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:09.780734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d58666bd-5cbnj" event={"ID":"3f53716a-dba1-452f-a392-66cec3b5d01e","Type":"ContainerStarted","Data":"0bde600f258f1fc5bcbbb57982118652cfa9d7d1bb2d03ad63da6aea17498425"}
Sep 09 20:26:09.781970 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:09.781949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-dzfqg" event={"ID":"a4ec44b7-98b7-4276-90cb-14e2766fc1a3","Type":"ContainerStarted","Data":"8e61cf203ffe2c637ba7e97046aa717a6a58227c2bf8268449848a737b1f4d70"}
Sep 09 20:26:10.789079 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:10.789039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d58666bd-5cbnj" event={"ID":"3f53716a-dba1-452f-a392-66cec3b5d01e","Type":"ContainerStarted","Data":"49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3"}
Sep 09 20:26:10.790771 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:10.790742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-dzfqg" event={"ID":"a4ec44b7-98b7-4276-90cb-14e2766fc1a3","Type":"ContainerStarted","Data":"642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb"}
Sep 09 20:26:10.790906 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:10.790779    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-dzfqg" podUID="a4ec44b7-98b7-4276-90cb-14e2766fc1a3" containerName="authorino" containerID="cri-o://642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb" gracePeriod=30
Sep 09 20:26:10.804934 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:10.804878    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7d58666bd-5cbnj" podStartSLOduration=2.321841497 podStartE2EDuration="2.80486129s" totalImagesPullingTime="483.019793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:10.802928142 +0000 UTC m=+592.296139140" watchObservedRunningTime="2026-09-09 20:26:10.80486129 +0000 UTC m=+592.298072478"
Sep 09 20:26:10.820404 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:10.820322    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-dzfqg" podStartSLOduration=2.187942414 podStartE2EDuration="2.820299837s" totalImagesPullingTime="632.357423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:10.819327275 +0000 UTC m=+592.312538260" watchObservedRunningTime="2026-09-09 20:26:10.820299837 +0000 UTC m=+592.313510823"
Sep 09 20:26:10.839264 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:10.839228    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zxgsp"]
Sep 09 20:26:10.839449 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:10.839422    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-zxgsp" podUID="e3f99c15-9c26-4ac1-b525-d8a8242932f7" containerName="authorino" containerID="cri-o://aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b" gracePeriod=30
Sep 09 20:26:10.931819 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:10.931788    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d7c9dff58-mnwpv"]
Sep 09 20:26:10.935033 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:10.935016    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d7c9dff58-mnwpv"
Sep 09 20:26:10.938158 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:10.938137    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 20:26:10.944951 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:10.944925    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d7c9dff58-mnwpv"]
Sep 09 20:26:11.031105 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.031068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9xw\" (UniqueName: \"kubernetes.io/projected/a1a785fb-2992-415c-8360-1b37115d04ee-kube-api-access-8m9xw\") pod \"authorino-d7c9dff58-mnwpv\" (UID: \"a1a785fb-2992-415c-8360-1b37115d04ee\") " pod="kuadrant-system/authorino-d7c9dff58-mnwpv"
Sep 09 20:26:11.031283 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.031159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1a785fb-2992-415c-8360-1b37115d04ee-tls-cert\") pod \"authorino-d7c9dff58-mnwpv\" (UID: \"a1a785fb-2992-415c-8360-1b37115d04ee\") " pod="kuadrant-system/authorino-d7c9dff58-mnwpv"
Sep 09 20:26:11.099417 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.099396    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dzfqg"
Sep 09 20:26:11.117619 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.117593    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zxgsp"
Sep 09 20:26:11.132381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.132334    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jh52\" (UniqueName: \"kubernetes.io/projected/a4ec44b7-98b7-4276-90cb-14e2766fc1a3-kube-api-access-5jh52\") pod \"a4ec44b7-98b7-4276-90cb-14e2766fc1a3\" (UID: \"a4ec44b7-98b7-4276-90cb-14e2766fc1a3\") "
Sep 09 20:26:11.132534 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.132443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlvqn\" (UniqueName: \"kubernetes.io/projected/e3f99c15-9c26-4ac1-b525-d8a8242932f7-kube-api-access-nlvqn\") pod \"e3f99c15-9c26-4ac1-b525-d8a8242932f7\" (UID: \"e3f99c15-9c26-4ac1-b525-d8a8242932f7\") "
Sep 09 20:26:11.132595 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.132572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9xw\" (UniqueName: \"kubernetes.io/projected/a1a785fb-2992-415c-8360-1b37115d04ee-kube-api-access-8m9xw\") pod \"authorino-d7c9dff58-mnwpv\" (UID: \"a1a785fb-2992-415c-8360-1b37115d04ee\") " pod="kuadrant-system/authorino-d7c9dff58-mnwpv"
Sep 09 20:26:11.132688 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.132668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1a785fb-2992-415c-8360-1b37115d04ee-tls-cert\") pod \"authorino-d7c9dff58-mnwpv\" (UID: \"a1a785fb-2992-415c-8360-1b37115d04ee\") " pod="kuadrant-system/authorino-d7c9dff58-mnwpv"
Sep 09 20:26:11.135195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.135171    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f99c15-9c26-4ac1-b525-d8a8242932f7-kube-api-access-nlvqn" (OuterVolumeSpecName: "kube-api-access-nlvqn") pod "e3f99c15-9c26-4ac1-b525-d8a8242932f7" (UID: "e3f99c15-9c26-4ac1-b525-d8a8242932f7"). InnerVolumeSpecName "kube-api-access-nlvqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:11.135195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.135183    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ec44b7-98b7-4276-90cb-14e2766fc1a3-kube-api-access-5jh52" (OuterVolumeSpecName: "kube-api-access-5jh52") pod "a4ec44b7-98b7-4276-90cb-14e2766fc1a3" (UID: "a4ec44b7-98b7-4276-90cb-14e2766fc1a3"). InnerVolumeSpecName "kube-api-access-5jh52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:11.135424 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.135228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1a785fb-2992-415c-8360-1b37115d04ee-tls-cert\") pod \"authorino-d7c9dff58-mnwpv\" (UID: \"a1a785fb-2992-415c-8360-1b37115d04ee\") " pod="kuadrant-system/authorino-d7c9dff58-mnwpv"
Sep 09 20:26:11.147326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.147303    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9xw\" (UniqueName: \"kubernetes.io/projected/a1a785fb-2992-415c-8360-1b37115d04ee-kube-api-access-8m9xw\") pod \"authorino-d7c9dff58-mnwpv\" (UID: \"a1a785fb-2992-415c-8360-1b37115d04ee\") " pod="kuadrant-system/authorino-d7c9dff58-mnwpv"
Sep 09 20:26:11.233289 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.233257    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jh52\" (UniqueName: \"kubernetes.io/projected/a4ec44b7-98b7-4276-90cb-14e2766fc1a3-kube-api-access-5jh52\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:26:11.233289 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.233285    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlvqn\" (UniqueName: \"kubernetes.io/projected/e3f99c15-9c26-4ac1-b525-d8a8242932f7-kube-api-access-nlvqn\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:26:11.245184 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.245159    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d7c9dff58-mnwpv"
Sep 09 20:26:11.521972 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.521946    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d7c9dff58-mnwpv"]
Sep 09 20:26:11.523275 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:26:11.523240    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1a785fb_2992_415c_8360_1b37115d04ee.slice/crio-840ae800b950eddd8300f0364caa6c5bfe0b85dab8abb904c2a43d3f2155f13b WatchSource:0}: Error finding container 840ae800b950eddd8300f0364caa6c5bfe0b85dab8abb904c2a43d3f2155f13b: Status 404 returned error can't find the container with id 840ae800b950eddd8300f0364caa6c5bfe0b85dab8abb904c2a43d3f2155f13b
Sep 09 20:26:11.798129 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.798028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d7c9dff58-mnwpv" event={"ID":"a1a785fb-2992-415c-8360-1b37115d04ee","Type":"ContainerStarted","Data":"840ae800b950eddd8300f0364caa6c5bfe0b85dab8abb904c2a43d3f2155f13b"}
Sep 09 20:26:11.799296 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.799272    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3f99c15-9c26-4ac1-b525-d8a8242932f7" containerID="aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b" exitCode=0
Sep 09 20:26:11.799410 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.799316    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-zxgsp"
Sep 09 20:26:11.799410 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.799372    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zxgsp" event={"ID":"e3f99c15-9c26-4ac1-b525-d8a8242932f7","Type":"ContainerDied","Data":"aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b"}
Sep 09 20:26:11.799410 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.799405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-zxgsp" event={"ID":"e3f99c15-9c26-4ac1-b525-d8a8242932f7","Type":"ContainerDied","Data":"145e22949e69cca3c4e04a88bac715ece9bd624204b15b734e827b2cd289297f"}
Sep 09 20:26:11.799596 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.799427    2581 scope.go:117] "RemoveContainer" containerID="aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b"
Sep 09 20:26:11.800656 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.800635    2581 generic.go:356] "Generic (PLEG): container finished" podID="a4ec44b7-98b7-4276-90cb-14e2766fc1a3" containerID="642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb" exitCode=0
Sep 09 20:26:11.800751 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.800707    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-dzfqg" event={"ID":"a4ec44b7-98b7-4276-90cb-14e2766fc1a3","Type":"ContainerDied","Data":"642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb"}
Sep 09 20:26:11.800751 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.800731    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-dzfqg"
Sep 09 20:26:11.800751 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.800739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-dzfqg" event={"ID":"a4ec44b7-98b7-4276-90cb-14e2766fc1a3","Type":"ContainerDied","Data":"8e61cf203ffe2c637ba7e97046aa717a6a58227c2bf8268449848a737b1f4d70"}
Sep 09 20:26:11.809593 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.809572    2581 scope.go:117] "RemoveContainer" containerID="aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b"
Sep 09 20:26:11.809934 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:26:11.809883    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b\": container with ID starting with aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b not found: ID does not exist" containerID="aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b"
Sep 09 20:26:11.809934 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.809908    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b"} err="failed to get container status \"aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b\": rpc error: code = NotFound desc = could not find container \"aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b\": container with ID starting with aa4adc65949ced05fa12cd87779dc23b0797679ccc8f3a5c35eb8eecde85be1b not found: ID does not exist"
Sep 09 20:26:11.809934 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.809928    2581 scope.go:117] "RemoveContainer" containerID="642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb"
Sep 09 20:26:11.818214 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.818198    2581 scope.go:117] "RemoveContainer" containerID="642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb"
Sep 09 20:26:11.818495 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:26:11.818476    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb\": container with ID starting with 642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb not found: ID does not exist" containerID="642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb"
Sep 09 20:26:11.818545 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.818502    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb"} err="failed to get container status \"642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb\": rpc error: code = NotFound desc = could not find container \"642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb\": container with ID starting with 642f87243952241ee13085034b7c959821bb131417ac515333cde69d0f1ca3cb not found: ID does not exist"
Sep 09 20:26:11.829552 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.829529    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zxgsp"]
Sep 09 20:26:11.836281 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.836260    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-zxgsp"]
Sep 09 20:26:11.857254 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.857142    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-dzfqg"]
Sep 09 20:26:11.859512 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:11.859492    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-dzfqg"]
Sep 09 20:26:12.807097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:12.807058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d7c9dff58-mnwpv" event={"ID":"a1a785fb-2992-415c-8360-1b37115d04ee","Type":"ContainerStarted","Data":"40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443"}
Sep 09 20:26:12.822821 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:12.822778    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d7c9dff58-mnwpv" podStartSLOduration=2.384813354 podStartE2EDuration="2.822761659s" totalImagesPullingTime="437.948305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:12.821719413 +0000 UTC m=+594.314930397" watchObservedRunningTime="2026-09-09 20:26:12.822761659 +0000 UTC m=+594.315972643"
Sep 09 20:26:12.849042 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:12.849010    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d58666bd-5cbnj"]
Sep 09 20:26:12.849208 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:12.849163    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7d58666bd-5cbnj" podUID="3f53716a-dba1-452f-a392-66cec3b5d01e" containerName="authorino" containerID="cri-o://49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3" gracePeriod=30
Sep 09 20:26:13.002885 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.002852    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4ec44b7-98b7-4276-90cb-14e2766fc1a3" path="/var/lib/kubelet/pods/a4ec44b7-98b7-4276-90cb-14e2766fc1a3/volumes"
Sep 09 20:26:13.003204 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.003191    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f99c15-9c26-4ac1-b525-d8a8242932f7" path="/var/lib/kubelet/pods/e3f99c15-9c26-4ac1-b525-d8a8242932f7/volumes"
Sep 09 20:26:13.095886 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.095861    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d58666bd-5cbnj"
Sep 09 20:26:13.149154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.149119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blfgb\" (UniqueName: \"kubernetes.io/projected/3f53716a-dba1-452f-a392-66cec3b5d01e-kube-api-access-blfgb\") pod \"3f53716a-dba1-452f-a392-66cec3b5d01e\" (UID: \"3f53716a-dba1-452f-a392-66cec3b5d01e\") "
Sep 09 20:26:13.151329 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.151297    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f53716a-dba1-452f-a392-66cec3b5d01e-kube-api-access-blfgb" (OuterVolumeSpecName: "kube-api-access-blfgb") pod "3f53716a-dba1-452f-a392-66cec3b5d01e" (UID: "3f53716a-dba1-452f-a392-66cec3b5d01e"). InnerVolumeSpecName "kube-api-access-blfgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:13.249895 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.249844    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blfgb\" (UniqueName: \"kubernetes.io/projected/3f53716a-dba1-452f-a392-66cec3b5d01e-kube-api-access-blfgb\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:26:13.820168 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.820126    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f53716a-dba1-452f-a392-66cec3b5d01e" containerID="49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3" exitCode=0
Sep 09 20:26:13.820640 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.820175    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d58666bd-5cbnj"
Sep 09 20:26:13.820640 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.820209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d58666bd-5cbnj" event={"ID":"3f53716a-dba1-452f-a392-66cec3b5d01e","Type":"ContainerDied","Data":"49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3"}
Sep 09 20:26:13.820640 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.820252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7d58666bd-5cbnj" event={"ID":"3f53716a-dba1-452f-a392-66cec3b5d01e","Type":"ContainerDied","Data":"0bde600f258f1fc5bcbbb57982118652cfa9d7d1bb2d03ad63da6aea17498425"}
Sep 09 20:26:13.820640 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.820274    2581 scope.go:117] "RemoveContainer" containerID="49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3"
Sep 09 20:26:13.829279 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.829261    2581 scope.go:117] "RemoveContainer" containerID="49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3"
Sep 09 20:26:13.829567 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:26:13.829547    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3\": container with ID starting with 49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3 not found: ID does not exist" containerID="49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3"
Sep 09 20:26:13.829641 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.829578    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3"} err="failed to get container status \"49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3\": rpc error: code = NotFound desc = could not find container \"49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3\": container with ID starting with 49502647d909149dda97d9fbb411c775c69c1cc0a646244586f78a1ad4712dc3 not found: ID does not exist"
Sep 09 20:26:13.843258 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.843238    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d58666bd-5cbnj"]
Sep 09 20:26:13.849300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:13.849280    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d58666bd-5cbnj"]
Sep 09 20:26:15.001448 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:15.001417    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f53716a-dba1-452f-a392-66cec3b5d01e" path="/var/lib/kubelet/pods/3f53716a-dba1-452f-a392-66cec3b5d01e/volumes"
Sep 09 20:26:18.945734 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:18.945710    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:26:18.947222 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:18.947200    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:26:52.239662 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.239620    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"]
Sep 09 20:26:52.242465 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.240097    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ec44b7-98b7-4276-90cb-14e2766fc1a3" containerName="authorino"
Sep 09 20:26:52.242465 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.240116    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ec44b7-98b7-4276-90cb-14e2766fc1a3" containerName="authorino"
Sep 09 20:26:52.242465 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.240136    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f99c15-9c26-4ac1-b525-d8a8242932f7" containerName="authorino"
Sep 09 20:26:52.242465 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.240141    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f99c15-9c26-4ac1-b525-d8a8242932f7" containerName="authorino"
Sep 09 20:26:52.242465 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.240150    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f53716a-dba1-452f-a392-66cec3b5d01e" containerName="authorino"
Sep 09 20:26:52.242465 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.240156    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f53716a-dba1-452f-a392-66cec3b5d01e" containerName="authorino"
Sep 09 20:26:52.242465 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.240208    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f99c15-9c26-4ac1-b525-d8a8242932f7" containerName="authorino"
Sep 09 20:26:52.242465 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.240213    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f53716a-dba1-452f-a392-66cec3b5d01e" containerName="authorino"
Sep 09 20:26:52.242465 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.240221    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ec44b7-98b7-4276-90cb-14e2766fc1a3" containerName="authorino"
Sep 09 20:26:52.243211 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.243183    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"
Sep 09 20:26:52.245809 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.245788    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hfrlt\""
Sep 09 20:26:52.245911 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.245817    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 20:26:52.252859 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.252835    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"]
Sep 09 20:26:52.273480 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.273449    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zdk65"]
Sep 09 20:26:52.276595 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.276580    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65"
Sep 09 20:26:52.288642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.288617    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zdk65"]
Sep 09 20:26:52.375526 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.375487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz77b\" (UniqueName: \"kubernetes.io/projected/63bec830-c2e4-4a4a-a998-280cbae1baf7-kube-api-access-cz77b\") pod \"payload-processing-5b54b7fbb7-zdk65\" (UID: \"63bec830-c2e4-4a4a-a998-280cbae1baf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65"
Sep 09 20:26:52.375707 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.375543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnnxp\" (UniqueName: \"kubernetes.io/projected/0e70b798-2892-471a-bf91-81797e3ff923-kube-api-access-mnnxp\") pod \"payload-pre-processing-59fbc47bf8-qzrnd\" (UID: \"0e70b798-2892-471a-bf91-81797e3ff923\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"
Sep 09 20:26:52.375707 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.375598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e70b798-2892-471a-bf91-81797e3ff923-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qzrnd\" (UID: \"0e70b798-2892-471a-bf91-81797e3ff923\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"
Sep 09 20:26:52.375707 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.375650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bec830-c2e4-4a4a-a998-280cbae1baf7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zdk65\" (UID: \"63bec830-c2e4-4a4a-a998-280cbae1baf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65"
Sep 09 20:26:52.476925 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.476881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnnxp\" (UniqueName: \"kubernetes.io/projected/0e70b798-2892-471a-bf91-81797e3ff923-kube-api-access-mnnxp\") pod \"payload-pre-processing-59fbc47bf8-qzrnd\" (UID: \"0e70b798-2892-471a-bf91-81797e3ff923\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"
Sep 09 20:26:52.477133 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.476938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e70b798-2892-471a-bf91-81797e3ff923-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qzrnd\" (UID: \"0e70b798-2892-471a-bf91-81797e3ff923\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"
Sep 09 20:26:52.477133 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.476985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bec830-c2e4-4a4a-a998-280cbae1baf7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zdk65\" (UID: \"63bec830-c2e4-4a4a-a998-280cbae1baf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65"
Sep 09 20:26:52.477133 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.477060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz77b\" (UniqueName: \"kubernetes.io/projected/63bec830-c2e4-4a4a-a998-280cbae1baf7-kube-api-access-cz77b\") pod \"payload-processing-5b54b7fbb7-zdk65\" (UID: \"63bec830-c2e4-4a4a-a998-280cbae1baf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65"
Sep 09 20:26:52.477891 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.477866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bec830-c2e4-4a4a-a998-280cbae1baf7-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-zdk65\" (UID: \"63bec830-c2e4-4a4a-a998-280cbae1baf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65"
Sep 09 20:26:52.477963 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.477905    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e70b798-2892-471a-bf91-81797e3ff923-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qzrnd\" (UID: \"0e70b798-2892-471a-bf91-81797e3ff923\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"
Sep 09 20:26:52.486282 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.486244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz77b\" (UniqueName: \"kubernetes.io/projected/63bec830-c2e4-4a4a-a998-280cbae1baf7-kube-api-access-cz77b\") pod \"payload-processing-5b54b7fbb7-zdk65\" (UID: \"63bec830-c2e4-4a4a-a998-280cbae1baf7\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65"
Sep 09 20:26:52.486473 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.486323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnnxp\" (UniqueName: \"kubernetes.io/projected/0e70b798-2892-471a-bf91-81797e3ff923-kube-api-access-mnnxp\") pod \"payload-pre-processing-59fbc47bf8-qzrnd\" (UID: \"0e70b798-2892-471a-bf91-81797e3ff923\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"
Sep 09 20:26:52.554271 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.554169    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"
Sep 09 20:26:52.587499 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.587468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65"
Sep 09 20:26:52.864242 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.864216    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"]
Sep 09 20:26:52.864400 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:26:52.864296    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e70b798_2892_471a_bf91_81797e3ff923.slice/crio-b8d03629ade4d2419a0edb00b5582debe258111cc7469b1cbf5bae3bbca74fbb WatchSource:0}: Error finding container b8d03629ade4d2419a0edb00b5582debe258111cc7469b1cbf5bae3bbca74fbb: Status 404 returned error can't find the container with id b8d03629ade4d2419a0edb00b5582debe258111cc7469b1cbf5bae3bbca74fbb
Sep 09 20:26:52.866141 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.866124    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:26:52.891902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:52.891877    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-zdk65"]
Sep 09 20:26:52.893792 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:26:52.893770    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bec830_c2e4_4a4a_a998_280cbae1baf7.slice/crio-dec46511f6bd3052a3e18e32e6c184a923d2fbb90a6c99f9021ce25b0c28387b WatchSource:0}: Error finding container dec46511f6bd3052a3e18e32e6c184a923d2fbb90a6c99f9021ce25b0c28387b: Status 404 returned error can't find the container with id dec46511f6bd3052a3e18e32e6c184a923d2fbb90a6c99f9021ce25b0c28387b
Sep 09 20:26:53.017908 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:53.017871    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd" event={"ID":"0e70b798-2892-471a-bf91-81797e3ff923","Type":"ContainerStarted","Data":"b8d03629ade4d2419a0edb00b5582debe258111cc7469b1cbf5bae3bbca74fbb"}
Sep 09 20:26:53.019044 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:53.019024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65" event={"ID":"63bec830-c2e4-4a4a-a998-280cbae1baf7","Type":"ContainerStarted","Data":"dec46511f6bd3052a3e18e32e6c184a923d2fbb90a6c99f9021ce25b0c28387b"}
Sep 09 20:26:55.033860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:55.033747    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd" event={"ID":"0e70b798-2892-471a-bf91-81797e3ff923","Type":"ContainerStarted","Data":"85c8b65de7280d9cff56d4960cf251966fad13fad10d555c6705366669c4c92d"}
Sep 09 20:26:55.033860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:55.033819    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"
Sep 09 20:26:55.035570 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:55.035537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65" event={"ID":"63bec830-c2e4-4a4a-a998-280cbae1baf7","Type":"ContainerStarted","Data":"bebf6e7962c1f744d2b7fd4ea18b22569cec5e0f043c2894f4ce611536d44c2c"}
Sep 09 20:26:55.035707 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:55.035589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65"
Sep 09 20:26:55.051600 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:55.051546    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd" podStartSLOduration=1.189280354 podStartE2EDuration="3.05153184s" totalImagesPullingTime="1.862251486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:55.050215992 +0000 UTC m=+636.543426977" watchObservedRunningTime="2026-09-09 20:26:55.05153184 +0000 UTC m=+636.544742824"
Sep 09 20:26:55.069119 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:26:55.069065    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65" podStartSLOduration=1.242955946 podStartE2EDuration="3.069052166s" totalImagesPullingTime="1.82609622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:55.068087283 +0000 UTC m=+636.561298273" watchObservedRunningTime="2026-09-09 20:26:55.069052166 +0000 UTC m=+636.562263149"
Sep 09 20:27:06.042972 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:06.042937    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qzrnd"
Sep 09 20:27:06.043390 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:06.043009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-zdk65"
Sep 09 20:27:57.151434 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.151400    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"]
Sep 09 20:27:57.155395 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.155372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.159511 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.159490    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 20:27:57.159640 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.159567    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xlwgd\""
Sep 09 20:27:57.159717 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.159570    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 20:27:57.159717 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.159585    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 20:27:57.169960 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.169934    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"]
Sep 09 20:27:57.308710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.308681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.308710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.308723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.308932 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.308764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.308932 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.308792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.308932 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.308894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.309044 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.308936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66nfq\" (UniqueName: \"kubernetes.io/projected/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-kube-api-access-66nfq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.410023 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.409939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.410023 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.409993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.410238 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.410031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.410238 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.410054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66nfq\" (UniqueName: \"kubernetes.io/projected/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-kube-api-access-66nfq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.410238 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.410092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.410238 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.410136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.410453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.410384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.410453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.410426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.410558 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.410498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.412630 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.412609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.412774 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.412757    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.417792 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.417774    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66nfq\" (UniqueName: \"kubernetes.io/projected/f1c7f0ec-985b-4045-8d39-c2682fc2a2cb-kube-api-access-66nfq\") pod \"e2e-trlp-test-simulated-kserve-598466548d-xwx4v\" (UID: \"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.472910 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.472883    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:27:57.745973 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:57.745943    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"]
Sep 09 20:27:57.747728 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:27:57.747696    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c7f0ec_985b_4045_8d39_c2682fc2a2cb.slice/crio-be99f854f8086fb19f7f9cebb2dcfbd0a568408fe55283f2622ce9bf9ab5692e WatchSource:0}: Error finding container be99f854f8086fb19f7f9cebb2dcfbd0a568408fe55283f2622ce9bf9ab5692e: Status 404 returned error can't find the container with id be99f854f8086fb19f7f9cebb2dcfbd0a568408fe55283f2622ce9bf9ab5692e
Sep 09 20:27:58.362835 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:58.362795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v" event={"ID":"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb","Type":"ContainerStarted","Data":"be99f854f8086fb19f7f9cebb2dcfbd0a568408fe55283f2622ce9bf9ab5692e"}
Sep 09 20:27:59.369732 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:59.369697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v" event={"ID":"f1c7f0ec-985b-4045-8d39-c2682fc2a2cb","Type":"ContainerStarted","Data":"78067d803cbe00c93cf753c77904aae8af8936168a34652d31d8db1d2b8756d3"}
Sep 09 20:27:59.386921 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:27:59.386869    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v" podStartSLOduration=1.3395483160000001 podStartE2EDuration="2.386852966s" totalImagesPullingTime="1.04730465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:27:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:27:59.385552193 +0000 UTC m=+700.878763177" watchObservedRunningTime="2026-09-09 20:27:59.386852966 +0000 UTC m=+700.880063950"
Sep 09 20:28:07.473648 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:07.473601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:28:07.474166 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:07.473676    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:28:07.486539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:07.486503    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:28:08.428096 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:08.428067    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-xwx4v"
Sep 09 20:28:16.900284 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:16.900202    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"]
Sep 09 20:28:16.903738 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:16.903714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:16.906576 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:16.906554    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 09 20:28:16.914453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:16.914432    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"]
Sep 09 20:28:16.968214 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:16.968188    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:16.968384 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:16.968220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:16.968384 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:16.968244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:16.968384 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:16.968262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66b99\" (UniqueName: \"kubernetes.io/projected/02255117-9878-4ac7-9f9b-5fc375a7cdab-kube-api-access-66b99\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:16.968497 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:16.968408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:16.968497 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:16.968439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02255117-9878-4ac7-9f9b-5fc375a7cdab-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.069159 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.069125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.069159 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.069160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02255117-9878-4ac7-9f9b-5fc375a7cdab-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.069448 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.069189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.069448 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.069206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.069448 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.069288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.069448 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.069328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66b99\" (UniqueName: \"kubernetes.io/projected/02255117-9878-4ac7-9f9b-5fc375a7cdab-kube-api-access-66b99\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.069666 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.069562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.069666 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.069627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.069787 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.069685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.071714 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.071695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02255117-9878-4ac7-9f9b-5fc375a7cdab-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.071991 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.071972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02255117-9878-4ac7-9f9b-5fc375a7cdab-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.082132 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.082114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66b99\" (UniqueName: \"kubernetes.io/projected/02255117-9878-4ac7-9f9b-5fc375a7cdab-kube-api-access-66b99\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5q5tk\" (UID: \"02255117-9878-4ac7-9f9b-5fc375a7cdab\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.214184 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.214150    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:17.485097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:17.485061    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"]
Sep 09 20:28:17.486522 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:28:17.486499    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02255117_9878_4ac7_9f9b_5fc375a7cdab.slice/crio-325d7952b09fc7ecb54579e41c68f1ebc7ca12d6f0bd9c27f6361e91341bd999 WatchSource:0}: Error finding container 325d7952b09fc7ecb54579e41c68f1ebc7ca12d6f0bd9c27f6361e91341bd999: Status 404 returned error can't find the container with id 325d7952b09fc7ecb54579e41c68f1ebc7ca12d6f0bd9c27f6361e91341bd999
Sep 09 20:28:18.469727 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:18.469685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk" event={"ID":"02255117-9878-4ac7-9f9b-5fc375a7cdab","Type":"ContainerStarted","Data":"8d5ffbb07836a3b49ff6a1f40c75806e6cbf08f15cc011fd45e231febc436602"}
Sep 09 20:28:18.470115 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:18.469749    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk" event={"ID":"02255117-9878-4ac7-9f9b-5fc375a7cdab","Type":"ContainerStarted","Data":"325d7952b09fc7ecb54579e41c68f1ebc7ca12d6f0bd9c27f6361e91341bd999"}
Sep 09 20:28:18.490055 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:18.490010    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk" podStartSLOduration=2.288505603 podStartE2EDuration="2.489994241s" totalImagesPullingTime="201.488638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:18.488949912 +0000 UTC m=+719.982160897" watchObservedRunningTime="2026-09-09 20:28:18.489994241 +0000 UTC m=+719.983205248"
Sep 09 20:28:27.214557 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:27.214515    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:27.214990 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:27.214598    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:27.227042 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:27.227011    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:27.527043 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:27.526959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5q5tk"
Sep 09 20:28:28.357980 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.357948    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"]
Sep 09 20:28:28.361751 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.361731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.364746 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.364728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 20:28:28.371117 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.371097    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"]
Sep 09 20:28:28.466838 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.466805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.466838 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.466839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.467056 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.466861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7lk\" (UniqueName: \"kubernetes.io/projected/8e634968-5d7e-42ec-ac1f-efe0037f1527-kube-api-access-fs7lk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.467056 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.466938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.467056 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.467010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e634968-5d7e-42ec-ac1f-efe0037f1527-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.467196 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.467086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.568093 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.568061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.568278 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.568122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e634968-5d7e-42ec-ac1f-efe0037f1527-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.568278 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.568165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.568278 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.568189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.568278 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.568204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.568278 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.568223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7lk\" (UniqueName: \"kubernetes.io/projected/8e634968-5d7e-42ec-ac1f-efe0037f1527-kube-api-access-fs7lk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.568597 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.568537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.568597 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.568576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.568668 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.568626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.570700 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.570671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e634968-5d7e-42ec-ac1f-efe0037f1527-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.570922 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.570903    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e634968-5d7e-42ec-ac1f-efe0037f1527-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.577503 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.577482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7lk\" (UniqueName: \"kubernetes.io/projected/8e634968-5d7e-42ec-ac1f-efe0037f1527-kube-api-access-fs7lk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h577s\" (UID: \"8e634968-5d7e-42ec-ac1f-efe0037f1527\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.671473 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.671391    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:28.943444 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:28.943420    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"]
Sep 09 20:28:28.946063 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:28:28.946034    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e634968_5d7e_42ec_ac1f_efe0037f1527.slice/crio-f476895ad3aaa64f5cae1c90b4383494f098e2f06254a87e7d74bb1317bac314 WatchSource:0}: Error finding container f476895ad3aaa64f5cae1c90b4383494f098e2f06254a87e7d74bb1317bac314: Status 404 returned error can't find the container with id f476895ad3aaa64f5cae1c90b4383494f098e2f06254a87e7d74bb1317bac314
Sep 09 20:28:29.528640 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:29.528545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s" event={"ID":"8e634968-5d7e-42ec-ac1f-efe0037f1527","Type":"ContainerStarted","Data":"c30d42250416ba0a0317893c632dcbc545b7172f0f4779f660efb7b62441a86e"}
Sep 09 20:28:29.528640 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:29.528584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s" event={"ID":"8e634968-5d7e-42ec-ac1f-efe0037f1527","Type":"ContainerStarted","Data":"f476895ad3aaa64f5cae1c90b4383494f098e2f06254a87e7d74bb1317bac314"}
Sep 09 20:28:29.546371 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:29.546296    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s" podStartSLOduration=1.341125557 podStartE2EDuration="1.546280317s" totalImagesPullingTime="205.15476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:29.54446717 +0000 UTC m=+731.037678155" watchObservedRunningTime="2026-09-09 20:28:29.546280317 +0000 UTC m=+731.039491300"
Sep 09 20:28:38.672223 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:38.672189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:38.672223 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:38.672231    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:38.684433 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:38.684411    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:28:39.593743 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:28:39.593714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h577s"
Sep 09 20:29:32.153860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.153822    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69b688bd6c-s297w"]
Sep 09 20:29:32.157064 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.157047    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69b688bd6c-s297w"
Sep 09 20:29:32.170208 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.170180    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69b688bd6c-s297w"]
Sep 09 20:29:32.218153 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.218103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25j9q\" (UniqueName: \"kubernetes.io/projected/308493ad-7d66-45a1-8b11-828830f7b3a1-kube-api-access-25j9q\") pod \"authorino-69b688bd6c-s297w\" (UID: \"308493ad-7d66-45a1-8b11-828830f7b3a1\") " pod="kuadrant-system/authorino-69b688bd6c-s297w"
Sep 09 20:29:32.218320 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.218239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/308493ad-7d66-45a1-8b11-828830f7b3a1-tls-cert\") pod \"authorino-69b688bd6c-s297w\" (UID: \"308493ad-7d66-45a1-8b11-828830f7b3a1\") " pod="kuadrant-system/authorino-69b688bd6c-s297w"
Sep 09 20:29:32.318944 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.318906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25j9q\" (UniqueName: \"kubernetes.io/projected/308493ad-7d66-45a1-8b11-828830f7b3a1-kube-api-access-25j9q\") pod \"authorino-69b688bd6c-s297w\" (UID: \"308493ad-7d66-45a1-8b11-828830f7b3a1\") " pod="kuadrant-system/authorino-69b688bd6c-s297w"
Sep 09 20:29:32.319118 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.318969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/308493ad-7d66-45a1-8b11-828830f7b3a1-tls-cert\") pod \"authorino-69b688bd6c-s297w\" (UID: \"308493ad-7d66-45a1-8b11-828830f7b3a1\") " pod="kuadrant-system/authorino-69b688bd6c-s297w"
Sep 09 20:29:32.321661 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.321630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/308493ad-7d66-45a1-8b11-828830f7b3a1-tls-cert\") pod \"authorino-69b688bd6c-s297w\" (UID: \"308493ad-7d66-45a1-8b11-828830f7b3a1\") " pod="kuadrant-system/authorino-69b688bd6c-s297w"
Sep 09 20:29:32.327252 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.327223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25j9q\" (UniqueName: \"kubernetes.io/projected/308493ad-7d66-45a1-8b11-828830f7b3a1-kube-api-access-25j9q\") pod \"authorino-69b688bd6c-s297w\" (UID: \"308493ad-7d66-45a1-8b11-828830f7b3a1\") " pod="kuadrant-system/authorino-69b688bd6c-s297w"
Sep 09 20:29:32.466418 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.466387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69b688bd6c-s297w"
Sep 09 20:29:32.740106 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.740076    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69b688bd6c-s297w"]
Sep 09 20:29:32.741996 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:29:32.741966    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod308493ad_7d66_45a1_8b11_828830f7b3a1.slice/crio-5e824eb58dadbb0f6e0fae682e322fe7f4c8d89efb9eca75d99c1f3980527d01 WatchSource:0}: Error finding container 5e824eb58dadbb0f6e0fae682e322fe7f4c8d89efb9eca75d99c1f3980527d01: Status 404 returned error can't find the container with id 5e824eb58dadbb0f6e0fae682e322fe7f4c8d89efb9eca75d99c1f3980527d01
Sep 09 20:29:32.860877 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:32.860841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69b688bd6c-s297w" event={"ID":"308493ad-7d66-45a1-8b11-828830f7b3a1","Type":"ContainerStarted","Data":"5e824eb58dadbb0f6e0fae682e322fe7f4c8d89efb9eca75d99c1f3980527d01"}
Sep 09 20:29:33.871225 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:33.871120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69b688bd6c-s297w" event={"ID":"308493ad-7d66-45a1-8b11-828830f7b3a1","Type":"ContainerStarted","Data":"2ca4d56a729fe821dd85d38004526d01a8d5efbb38d58d9aca41819953b2a101"}
Sep 09 20:29:33.887980 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:33.887928    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-69b688bd6c-s297w" podStartSLOduration=1.015094214 podStartE2EDuration="1.887910589s" totalImagesPullingTime="872.816375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:29:33.887377755 +0000 UTC m=+795.380588742" watchObservedRunningTime="2026-09-09 20:29:33.887910589 +0000 UTC m=+795.381121572"
Sep 09 20:29:33.912510 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:33.912474    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d7c9dff58-mnwpv"]
Sep 09 20:29:33.912836 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:33.912796    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-d7c9dff58-mnwpv" podUID="a1a785fb-2992-415c-8360-1b37115d04ee" containerName="authorino" containerID="cri-o://40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443" gracePeriod=30
Sep 09 20:29:34.164435 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.164408    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d7c9dff58-mnwpv"
Sep 09 20:29:34.235547 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.235459    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m9xw\" (UniqueName: \"kubernetes.io/projected/a1a785fb-2992-415c-8360-1b37115d04ee-kube-api-access-8m9xw\") pod \"a1a785fb-2992-415c-8360-1b37115d04ee\" (UID: \"a1a785fb-2992-415c-8360-1b37115d04ee\") "
Sep 09 20:29:34.235547 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.235515    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1a785fb-2992-415c-8360-1b37115d04ee-tls-cert\") pod \"a1a785fb-2992-415c-8360-1b37115d04ee\" (UID: \"a1a785fb-2992-415c-8360-1b37115d04ee\") "
Sep 09 20:29:34.237891 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.237859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a785fb-2992-415c-8360-1b37115d04ee-kube-api-access-8m9xw" (OuterVolumeSpecName: "kube-api-access-8m9xw") pod "a1a785fb-2992-415c-8360-1b37115d04ee" (UID: "a1a785fb-2992-415c-8360-1b37115d04ee"). InnerVolumeSpecName "kube-api-access-8m9xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:29:34.247049 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.247022    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1a785fb-2992-415c-8360-1b37115d04ee-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a1a785fb-2992-415c-8360-1b37115d04ee" (UID: "a1a785fb-2992-415c-8360-1b37115d04ee"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:29:34.336646 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.336607    2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1a785fb-2992-415c-8360-1b37115d04ee-tls-cert\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:29:34.336646 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.336640    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m9xw\" (UniqueName: \"kubernetes.io/projected/a1a785fb-2992-415c-8360-1b37115d04ee-kube-api-access-8m9xw\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:29:34.877605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.877507    2581 generic.go:356] "Generic (PLEG): container finished" podID="a1a785fb-2992-415c-8360-1b37115d04ee" containerID="40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443" exitCode=0
Sep 09 20:29:34.877605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.877552    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d7c9dff58-mnwpv"
Sep 09 20:29:34.877605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.877593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d7c9dff58-mnwpv" event={"ID":"a1a785fb-2992-415c-8360-1b37115d04ee","Type":"ContainerDied","Data":"40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443"}
Sep 09 20:29:34.878125 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.877634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d7c9dff58-mnwpv" event={"ID":"a1a785fb-2992-415c-8360-1b37115d04ee","Type":"ContainerDied","Data":"840ae800b950eddd8300f0364caa6c5bfe0b85dab8abb904c2a43d3f2155f13b"}
Sep 09 20:29:34.878125 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.877656    2581 scope.go:117] "RemoveContainer" containerID="40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443"
Sep 09 20:29:34.888086 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.888068    2581 scope.go:117] "RemoveContainer" containerID="40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443"
Sep 09 20:29:34.888406 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:29:34.888384    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443\": container with ID starting with 40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443 not found: ID does not exist" containerID="40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443"
Sep 09 20:29:34.888525 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.888418    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443"} err="failed to get container status \"40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443\": rpc error: code = NotFound desc = could not find container \"40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443\": container with ID starting with 40f62011ce0ca4bbcf99bf5ef74773e2dcf127e9f82c3d5b193e9ce3ec146443 not found: ID does not exist"
Sep 09 20:29:34.899094 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.899060    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d7c9dff58-mnwpv"]
Sep 09 20:29:34.901539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:34.901515    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d7c9dff58-mnwpv"]
Sep 09 20:29:35.002962 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:35.002930    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1a785fb-2992-415c-8360-1b37115d04ee" path="/var/lib/kubelet/pods/a1a785fb-2992-415c-8360-1b37115d04ee/volumes"
Sep 09 20:29:57.257236 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.257203    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7589-75cd6a"]
Sep 09 20:29:57.257633 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.257593    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1a785fb-2992-415c-8360-1b37115d04ee" containerName="authorino"
Sep 09 20:29:57.257633 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.257607    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1a785fb-2992-415c-8360-1b37115d04ee" containerName="authorino"
Sep 09 20:29:57.257715 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.257671    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1a785fb-2992-415c-8360-1b37115d04ee" containerName="authorino"
Sep 09 20:29:57.260646 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.260627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7589-75cd6a"
Sep 09 20:29:57.263195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.263172    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5fctx\""
Sep 09 20:29:57.266647 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.266625    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7589-75cd6a"]
Sep 09 20:29:57.449205 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.449170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8f6\" (UniqueName: \"kubernetes.io/projected/48be8778-2115-433b-a895-49f1eac57cad-kube-api-access-6v8f6\") pod \"test-curl-7589-75cd6a\" (UID: \"48be8778-2115-433b-a895-49f1eac57cad\") " pod="opendatahub/test-curl-7589-75cd6a"
Sep 09 20:29:57.550027 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.549945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8f6\" (UniqueName: \"kubernetes.io/projected/48be8778-2115-433b-a895-49f1eac57cad-kube-api-access-6v8f6\") pod \"test-curl-7589-75cd6a\" (UID: \"48be8778-2115-433b-a895-49f1eac57cad\") " pod="opendatahub/test-curl-7589-75cd6a"
Sep 09 20:29:57.558133 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.558100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8f6\" (UniqueName: \"kubernetes.io/projected/48be8778-2115-433b-a895-49f1eac57cad-kube-api-access-6v8f6\") pod \"test-curl-7589-75cd6a\" (UID: \"48be8778-2115-433b-a895-49f1eac57cad\") " pod="opendatahub/test-curl-7589-75cd6a"
Sep 09 20:29:57.571006 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.570981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7589-75cd6a"
Sep 09 20:29:57.843126 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:57.843102    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7589-75cd6a"]
Sep 09 20:29:57.844764 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:29:57.844739    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48be8778_2115_433b_a895_49f1eac57cad.slice/crio-c76960b121823a06ec964d4e953a1968f8a16da2c4a83df7475526eabc6b6738 WatchSource:0}: Error finding container c76960b121823a06ec964d4e953a1968f8a16da2c4a83df7475526eabc6b6738: Status 404 returned error can't find the container with id c76960b121823a06ec964d4e953a1968f8a16da2c4a83df7475526eabc6b6738
Sep 09 20:29:58.000620 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:29:58.000587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7589-75cd6a" event={"ID":"48be8778-2115-433b-a895-49f1eac57cad","Type":"ContainerStarted","Data":"c76960b121823a06ec964d4e953a1968f8a16da2c4a83df7475526eabc6b6738"}
Sep 09 20:30:00.134288 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.134249    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg"]
Sep 09 20:30:00.145425 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.145396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg"]
Sep 09 20:30:00.145901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.145449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg"
Sep 09 20:30:00.147915 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.147889    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:30:00.159834 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.159789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg"
Sep 09 20:30:00.543162 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.543124    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg"]
Sep 09 20:30:00.544681 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:30:00.544638    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffaa8aa_2715_4a27_889f_12f211980ed5.slice/crio-36bd3e83fea30468cd9fec73b05df92a5cec5f7f6d279d861879933e90b350d6 WatchSource:0}: Error finding container 36bd3e83fea30468cd9fec73b05df92a5cec5f7f6d279d861879933e90b350d6: Status 404 returned error can't find the container with id 36bd3e83fea30468cd9fec73b05df92a5cec5f7f6d279d861879933e90b350d6
Sep 09 20:30:00.587153 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.587114    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"]
Sep 09 20:30:00.590783 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.590759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:30:00.593331 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.593310    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-21117b\""
Sep 09 20:30:00.593440 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.593312    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-21117b-dockercfg-655nq\""
Sep 09 20:30:00.601599 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.601579    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"]
Sep 09 20:30:00.618408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.618377    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"]
Sep 09 20:30:00.621962 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.621945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:30:00.632158 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.632128    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"]
Sep 09 20:30:00.677076 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.677041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9b9f9f-aa25-4398-ae65-abf3a644f103-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx\" (UID: \"cc9b9f9f-aa25-4398-ae65-abf3a644f103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:30:00.677263 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.677163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c683389c-4524-41e6-9501-a649e694a6d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-21117b-747669564f-vgc5h\" (UID: \"c683389c-4524-41e6-9501-a649e694a6d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:30:00.677263 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.677239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk6tm\" (UniqueName: \"kubernetes.io/projected/cc9b9f9f-aa25-4398-ae65-abf3a644f103-kube-api-access-sk6tm\") pod \"payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx\" (UID: \"cc9b9f9f-aa25-4398-ae65-abf3a644f103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:30:00.677388 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.677291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kjn2\" (UniqueName: \"kubernetes.io/projected/c683389c-4524-41e6-9501-a649e694a6d6-kube-api-access-8kjn2\") pod \"payload-processing-e2e-inf-a-21117b-747669564f-vgc5h\" (UID: \"c683389c-4524-41e6-9501-a649e694a6d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:30:00.778124 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.778086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c683389c-4524-41e6-9501-a649e694a6d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-21117b-747669564f-vgc5h\" (UID: \"c683389c-4524-41e6-9501-a649e694a6d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:30:00.778318 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.778143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk6tm\" (UniqueName: \"kubernetes.io/projected/cc9b9f9f-aa25-4398-ae65-abf3a644f103-kube-api-access-sk6tm\") pod \"payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx\" (UID: \"cc9b9f9f-aa25-4398-ae65-abf3a644f103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:30:00.778318 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.778165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kjn2\" (UniqueName: \"kubernetes.io/projected/c683389c-4524-41e6-9501-a649e694a6d6-kube-api-access-8kjn2\") pod \"payload-processing-e2e-inf-a-21117b-747669564f-vgc5h\" (UID: \"c683389c-4524-41e6-9501-a649e694a6d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:30:00.778318 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.778196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9b9f9f-aa25-4398-ae65-abf3a644f103-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx\" (UID: \"cc9b9f9f-aa25-4398-ae65-abf3a644f103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:30:00.778925 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.778900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9b9f9f-aa25-4398-ae65-abf3a644f103-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx\" (UID: \"cc9b9f9f-aa25-4398-ae65-abf3a644f103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:30:00.779025 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.778902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c683389c-4524-41e6-9501-a649e694a6d6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-21117b-747669564f-vgc5h\" (UID: \"c683389c-4524-41e6-9501-a649e694a6d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:30:00.786671 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.786642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk6tm\" (UniqueName: \"kubernetes.io/projected/cc9b9f9f-aa25-4398-ae65-abf3a644f103-kube-api-access-sk6tm\") pod \"payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx\" (UID: \"cc9b9f9f-aa25-4398-ae65-abf3a644f103\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:30:00.786798 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.786741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kjn2\" (UniqueName: \"kubernetes.io/projected/c683389c-4524-41e6-9501-a649e694a6d6-kube-api-access-8kjn2\") pod \"payload-processing-e2e-inf-a-21117b-747669564f-vgc5h\" (UID: \"c683389c-4524-41e6-9501-a649e694a6d6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:30:00.902816 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.902780    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:30:00.933525 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:00.933462    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:30:01.021270 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:01.021241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7589-75cd6a" event={"ID":"48be8778-2115-433b-a895-49f1eac57cad","Type":"ContainerStarted","Data":"5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab"}
Sep 09 20:30:01.022785 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:01.022748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg" event={"ID":"effaa8aa-2715-4a27-889f-12f211980ed5","Type":"ContainerStarted","Data":"36bd3e83fea30468cd9fec73b05df92a5cec5f7f6d279d861879933e90b350d6"}
Sep 09 20:30:01.037053 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:01.037010    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7589-75cd6a" podStartSLOduration=1.5463624299999998 podStartE2EDuration="4.036994243s" totalImagesPullingTime="2.490631813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:01.034803811 +0000 UTC m=+822.528014796" watchObservedRunningTime="2026-09-09 20:30:01.036994243 +0000 UTC m=+822.530205226"
Sep 09 20:30:01.178722 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:01.178696    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"]
Sep 09 20:30:01.180128 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:30:01.180100    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9b9f9f_aa25_4398_ae65_abf3a644f103.slice/crio-5689fd63c9beb6365a86f04d654d33c10d5b767d95c6777cc4d7e9217ae55f1c WatchSource:0}: Error finding container 5689fd63c9beb6365a86f04d654d33c10d5b767d95c6777cc4d7e9217ae55f1c: Status 404 returned error can't find the container with id 5689fd63c9beb6365a86f04d654d33c10d5b767d95c6777cc4d7e9217ae55f1c
Sep 09 20:30:01.213864 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:01.213842    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"]
Sep 09 20:30:01.215981 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:30:01.215951    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc683389c_4524_41e6_9501_a649e694a6d6.slice/crio-8ddd88e38f2942a2def325588851f6b156df3d72d8569354e4d42913aa8d0171 WatchSource:0}: Error finding container 8ddd88e38f2942a2def325588851f6b156df3d72d8569354e4d42913aa8d0171: Status 404 returned error can't find the container with id 8ddd88e38f2942a2def325588851f6b156df3d72d8569354e4d42913aa8d0171
Sep 09 20:30:01.613203 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:01.613120    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7589-75cd6a"]
Sep 09 20:30:01.616971 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:01.616946    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7589-75cd6a"]
Sep 09 20:30:02.031834 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:02.031786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h" event={"ID":"c683389c-4524-41e6-9501-a649e694a6d6","Type":"ContainerStarted","Data":"303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3"}
Sep 09 20:30:02.032031 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:02.031853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h" event={"ID":"c683389c-4524-41e6-9501-a649e694a6d6","Type":"ContainerStarted","Data":"8ddd88e38f2942a2def325588851f6b156df3d72d8569354e4d42913aa8d0171"}
Sep 09 20:30:02.032031 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:02.031901    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:30:02.034213 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:02.034124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx" event={"ID":"cc9b9f9f-aa25-4398-ae65-abf3a644f103","Type":"ContainerStarted","Data":"263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f"}
Sep 09 20:30:02.034213 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:02.034174    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx" event={"ID":"cc9b9f9f-aa25-4398-ae65-abf3a644f103","Type":"ContainerStarted","Data":"5689fd63c9beb6365a86f04d654d33c10d5b767d95c6777cc4d7e9217ae55f1c"}
Sep 09 20:30:02.034463 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:02.034241    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:30:02.034463 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:30:02.034302    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7589-75cd6a" container="test-curl-7589-75cd6a"
Sep 09 20:30:02.051006 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:02.050956    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h" podStartSLOduration=2.050940853 podStartE2EDuration="2.050940853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:02.047984063 +0000 UTC m=+823.541195043" watchObservedRunningTime="2026-09-09 20:30:02.050940853 +0000 UTC m=+823.544151837"
Sep 09 20:30:02.064829 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:02.064614    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx" podStartSLOduration=2.064595401 podStartE2EDuration="2.064595401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:02.063237902 +0000 UTC m=+823.556448887" watchObservedRunningTime="2026-09-09 20:30:02.064595401 +0000 UTC m=+823.557806384"
Sep 09 20:30:03.040190 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:03.040144    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7589-75cd6a" podUID="48be8778-2115-433b-a895-49f1eac57cad" containerName="test-curl-7589-75cd6a" containerID="cri-o://5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab" gracePeriod=2
Sep 09 20:30:04.046576 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:04.046538    2581 generic.go:356] "Generic (PLEG): container finished" podID="effaa8aa-2715-4a27-889f-12f211980ed5" containerID="edea9d0a74f89babe0c7934951688eeac4b38bd90c481fab984eb0fcaf1d5504" exitCode=7
Sep 09 20:30:04.046980 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:04.046602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg" event={"ID":"effaa8aa-2715-4a27-889f-12f211980ed5","Type":"ContainerDied","Data":"edea9d0a74f89babe0c7934951688eeac4b38bd90c481fab984eb0fcaf1d5504"}
Sep 09 20:30:04.046980 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:04.046792    2581 scope.go:117] "RemoveContainer" containerID="edea9d0a74f89babe0c7934951688eeac4b38bd90c481fab984eb0fcaf1d5504"
Sep 09 20:30:05.053574 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:05.053408    2581 generic.go:356] "Generic (PLEG): container finished" podID="effaa8aa-2715-4a27-889f-12f211980ed5" containerID="8eb56f3cc023fc4459612fe7d508c2c35281d06ef3d3ab4c2d25f72fe56676c8" exitCode=7
Sep 09 20:30:05.053574 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:05.053496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg" event={"ID":"effaa8aa-2715-4a27-889f-12f211980ed5","Type":"ContainerDied","Data":"8eb56f3cc023fc4459612fe7d508c2c35281d06ef3d3ab4c2d25f72fe56676c8"}
Sep 09 20:30:05.053574 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:05.053558    2581 scope.go:117] "RemoveContainer" containerID="edea9d0a74f89babe0c7934951688eeac4b38bd90c481fab984eb0fcaf1d5504"
Sep 09 20:30:05.053995 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:05.053632    2581 scope.go:117] "RemoveContainer" containerID="8eb56f3cc023fc4459612fe7d508c2c35281d06ef3d3ab4c2d25f72fe56676c8"
Sep 09 20:30:05.053995 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:30:05.053878    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg_odh-ai-gateway-infra(effaa8aa-2715-4a27-889f-12f211980ed5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg" podUID="effaa8aa-2715-4a27-889f-12f211980ed5"
Sep 09 20:30:05.225118 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:05.225094    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7589-75cd6a"
Sep 09 20:30:05.314552 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:05.314454    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v8f6\" (UniqueName: \"kubernetes.io/projected/48be8778-2115-433b-a895-49f1eac57cad-kube-api-access-6v8f6\") pod \"48be8778-2115-433b-a895-49f1eac57cad\" (UID: \"48be8778-2115-433b-a895-49f1eac57cad\") "
Sep 09 20:30:05.316788 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:05.316753    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48be8778-2115-433b-a895-49f1eac57cad-kube-api-access-6v8f6" (OuterVolumeSpecName: "kube-api-access-6v8f6") pod "48be8778-2115-433b-a895-49f1eac57cad" (UID: "48be8778-2115-433b-a895-49f1eac57cad"). InnerVolumeSpecName "kube-api-access-6v8f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:05.415423 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:05.415383    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v8f6\" (UniqueName: \"kubernetes.io/projected/48be8778-2115-433b-a895-49f1eac57cad-kube-api-access-6v8f6\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:30:06.061134 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:06.061102    2581 generic.go:356] "Generic (PLEG): container finished" podID="48be8778-2115-433b-a895-49f1eac57cad" containerID="5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab" exitCode=137
Sep 09 20:30:06.061640 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:06.061147    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7589-75cd6a"
Sep 09 20:30:06.061640 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:06.061188    2581 scope.go:117] "RemoveContainer" containerID="5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab"
Sep 09 20:30:06.063766 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:06.063731    2581 scope.go:117] "RemoveContainer" containerID="8eb56f3cc023fc4459612fe7d508c2c35281d06ef3d3ab4c2d25f72fe56676c8"
Sep 09 20:30:06.064625 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:30:06.063973    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg_odh-ai-gateway-infra(effaa8aa-2715-4a27-889f-12f211980ed5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg" podUID="effaa8aa-2715-4a27-889f-12f211980ed5"
Sep 09 20:30:06.072150 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:06.072110    2581 scope.go:117] "RemoveContainer" containerID="5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab"
Sep 09 20:30:06.072481 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:30:06.072462    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab\": container with ID starting with 5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab not found: ID does not exist" containerID="5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab"
Sep 09 20:30:06.072528 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:06.072490    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab"} err="failed to get container status \"5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab\": rpc error: code = NotFound desc = could not find container \"5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab\": container with ID starting with 5786225496b82aa87685d934314192dbb764766778f0d3ea339c014af634c9ab not found: ID does not exist"
Sep 09 20:30:06.084667 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:30:06.084636    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7589-75cd6a\" is forbidden: User \"system:node:ip-10-0-135-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-122.ec2.internal' and this object" podUID="48be8778-2115-433b-a895-49f1eac57cad" pod="opendatahub/test-curl-7589-75cd6a"
Sep 09 20:30:07.002202 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.002166    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48be8778-2115-433b-a895-49f1eac57cad" path="/var/lib/kubelet/pods/48be8778-2115-433b-a895-49f1eac57cad/volumes"
Sep 09 20:30:07.687775 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.687740    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"]
Sep 09 20:30:07.688158 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.688122    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48be8778-2115-433b-a895-49f1eac57cad" containerName="test-curl-7589-75cd6a"
Sep 09 20:30:07.688158 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.688134    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48be8778-2115-433b-a895-49f1eac57cad" containerName="test-curl-7589-75cd6a"
Sep 09 20:30:07.688244 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.688199    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48be8778-2115-433b-a895-49f1eac57cad" containerName="test-curl-7589-75cd6a"
Sep 09 20:30:07.692661 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.692643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:07.695160 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.695135    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-87ac1d4f-dockercfg-ztvzk\""
Sep 09 20:30:07.695160 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.695150    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-87ac1d4f\""
Sep 09 20:30:07.701288 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.701268    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"]
Sep 09 20:30:07.720859 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.720832    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"]
Sep 09 20:30:07.724290 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.724272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:07.734359 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.734314    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"]
Sep 09 20:30:07.737063 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.737038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-plugins-config-volume\") pod \"payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d\" (UID: \"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:07.737192 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.737099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2\" (UID: \"5f3164af-85f3-4dcb-ab78-a6722bc73b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:07.737192 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.737147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqkx\" (UniqueName: \"kubernetes.io/projected/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-kube-api-access-9gqkx\") pod \"payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2\" (UID: \"5f3164af-85f3-4dcb-ab78-a6722bc73b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:07.737292 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.737206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v77g7\" (UniqueName: \"kubernetes.io/projected/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-kube-api-access-v77g7\") pod \"payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d\" (UID: \"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:07.837881 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.837845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-plugins-config-volume\") pod \"payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d\" (UID: \"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:07.838060 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.837905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2\" (UID: \"5f3164af-85f3-4dcb-ab78-a6722bc73b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:07.838060 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.837924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqkx\" (UniqueName: \"kubernetes.io/projected/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-kube-api-access-9gqkx\") pod \"payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2\" (UID: \"5f3164af-85f3-4dcb-ab78-a6722bc73b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:07.838060 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.837953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77g7\" (UniqueName: \"kubernetes.io/projected/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-kube-api-access-v77g7\") pod \"payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d\" (UID: \"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:07.838678 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.838656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-plugins-config-volume\") pod \"payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d\" (UID: \"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:07.838718 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.838672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2\" (UID: \"5f3164af-85f3-4dcb-ab78-a6722bc73b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:07.849397 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.849365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqkx\" (UniqueName: \"kubernetes.io/projected/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-kube-api-access-9gqkx\") pod \"payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2\" (UID: \"5f3164af-85f3-4dcb-ab78-a6722bc73b5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:07.852942 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:07.852921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v77g7\" (UniqueName: \"kubernetes.io/projected/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-kube-api-access-v77g7\") pod \"payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d\" (UID: \"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:08.003730 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:08.003638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:08.035379 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:08.035334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:08.283743 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:30:08.283713    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3164af_85f3_4dcb_ab78_a6722bc73b5b.slice/crio-7f0a3c881acd7b1faa9ed4229cdea768344e5a407b96c28516690d446154b7e8 WatchSource:0}: Error finding container 7f0a3c881acd7b1faa9ed4229cdea768344e5a407b96c28516690d446154b7e8: Status 404 returned error can't find the container with id 7f0a3c881acd7b1faa9ed4229cdea768344e5a407b96c28516690d446154b7e8
Sep 09 20:30:08.284065 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:08.284032    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"]
Sep 09 20:30:08.320747 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:08.320719    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"]
Sep 09 20:30:08.322182 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:30:08.322141    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4e0920_5b18_4a8b_bdeb_06d154ed6aca.slice/crio-7a869ff7fe0e75be49f2f74d95df6395ea22ae48839cc8704925f61568152056 WatchSource:0}: Error finding container 7a869ff7fe0e75be49f2f74d95df6395ea22ae48839cc8704925f61568152056: Status 404 returned error can't find the container with id 7a869ff7fe0e75be49f2f74d95df6395ea22ae48839cc8704925f61568152056
Sep 09 20:30:09.085470 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:09.085436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2" event={"ID":"5f3164af-85f3-4dcb-ab78-a6722bc73b5b","Type":"ContainerStarted","Data":"c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3"}
Sep 09 20:30:09.085907 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:09.085487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2" event={"ID":"5f3164af-85f3-4dcb-ab78-a6722bc73b5b","Type":"ContainerStarted","Data":"7f0a3c881acd7b1faa9ed4229cdea768344e5a407b96c28516690d446154b7e8"}
Sep 09 20:30:09.085907 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:09.085536    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:09.087032 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:09.087006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d" event={"ID":"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca","Type":"ContainerStarted","Data":"e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f"}
Sep 09 20:30:09.087149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:09.087054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d" event={"ID":"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca","Type":"ContainerStarted","Data":"7a869ff7fe0e75be49f2f74d95df6395ea22ae48839cc8704925f61568152056"}
Sep 09 20:30:09.087149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:09.087080    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:09.102406 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:09.102324    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2" podStartSLOduration=2.102310331 podStartE2EDuration="2.102310331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:09.101107909 +0000 UTC m=+830.594318890" watchObservedRunningTime="2026-09-09 20:30:09.102310331 +0000 UTC m=+830.595521315"
Sep 09 20:30:09.117802 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:09.117757    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d" podStartSLOduration=2.11774203 podStartE2EDuration="2.11774203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:09.116524379 +0000 UTC m=+830.609735364" watchObservedRunningTime="2026-09-09 20:30:09.11774203 +0000 UTC m=+830.610953013"
Sep 09 20:30:12.125466 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.125430    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"]
Sep 09 20:30:12.129034 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.129018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:30:12.131386 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.131366    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-812760-dockercfg-zr6tq\""
Sep 09 20:30:12.131386 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.131372    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-812760\""
Sep 09 20:30:12.139252 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.139224    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"]
Sep 09 20:30:12.176674 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.176634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkmr5\" (UniqueName: \"kubernetes.io/projected/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-kube-api-access-dkmr5\") pod \"payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw\" (UID: \"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156\") " pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:30:12.176674 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.176677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-plugins-config-volume\") pod \"payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw\" (UID: \"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156\") " pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:30:12.277262 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.277222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkmr5\" (UniqueName: \"kubernetes.io/projected/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-kube-api-access-dkmr5\") pod \"payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw\" (UID: \"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156\") " pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:30:12.277262 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.277259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-plugins-config-volume\") pod \"payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw\" (UID: \"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156\") " pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:30:12.278004 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.277979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-plugins-config-volume\") pod \"payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw\" (UID: \"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156\") " pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:30:12.286258 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.286228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkmr5\" (UniqueName: \"kubernetes.io/projected/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-kube-api-access-dkmr5\") pod \"payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw\" (UID: \"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156\") " pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:30:12.440979 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.440931    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:30:12.712937 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:12.712910    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"]
Sep 09 20:30:12.714444 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:30:12.714411    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6931e6_f18f_45c4_a2fd_bdd3a7ecb156.slice/crio-e4fd213cb7d6550eaed07d3b8eb13e8af7bca881db0f09c997e12af52706e6b1 WatchSource:0}: Error finding container e4fd213cb7d6550eaed07d3b8eb13e8af7bca881db0f09c997e12af52706e6b1: Status 404 returned error can't find the container with id e4fd213cb7d6550eaed07d3b8eb13e8af7bca881db0f09c997e12af52706e6b1
Sep 09 20:30:13.042563 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:13.042486    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:30:13.042563 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:13.042558    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:30:13.112901 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:13.112868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw" event={"ID":"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156","Type":"ContainerStarted","Data":"891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984"}
Sep 09 20:30:13.113061 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:13.112927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw" event={"ID":"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156","Type":"ContainerStarted","Data":"e4fd213cb7d6550eaed07d3b8eb13e8af7bca881db0f09c997e12af52706e6b1"}
Sep 09 20:30:13.113061 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:13.112956    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:30:13.132708 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:13.132657    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw" podStartSLOduration=1.132639981 podStartE2EDuration="1.132639981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:13.129612925 +0000 UTC m=+834.622823910" watchObservedRunningTime="2026-09-09 20:30:13.132639981 +0000 UTC m=+834.625851022"
Sep 09 20:30:16.997033 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:16.997004    2581 scope.go:117] "RemoveContainer" containerID="8eb56f3cc023fc4459612fe7d508c2c35281d06ef3d3ab4c2d25f72fe56676c8"
Sep 09 20:30:18.148306 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:18.148273    2581 generic.go:356] "Generic (PLEG): container finished" podID="effaa8aa-2715-4a27-889f-12f211980ed5" containerID="5613878637debb84fbe8d9d3eeb7d87311a533b4b9c5ff49a67885356d2382a8" exitCode=0
Sep 09 20:30:18.148730 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:18.148381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg" event={"ID":"effaa8aa-2715-4a27-889f-12f211980ed5","Type":"ContainerDied","Data":"5613878637debb84fbe8d9d3eeb7d87311a533b4b9c5ff49a67885356d2382a8"}
Sep 09 20:30:18.148730 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:18.148484    2581 scope.go:117] "RemoveContainer" containerID="8eb56f3cc023fc4459612fe7d508c2c35281d06ef3d3ab4c2d25f72fe56676c8"
Sep 09 20:30:19.177395 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:19.177366    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg"]
Sep 09 20:30:19.299435 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:19.299368    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg"
Sep 09 20:30:20.094929 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:20.094894    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:20.095120 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:20.095105    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:20.163984 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:20.163956    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg"
Sep 09 20:30:20.163984 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:20.163969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg" event={"ID":"effaa8aa-2715-4a27-889f-12f211980ed5","Type":"ContainerDied","Data":"36bd3e83fea30468cd9fec73b05df92a5cec5f7f6d279d861879933e90b350d6"}
Sep 09 20:30:20.164227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:20.164016    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36bd3e83fea30468cd9fec73b05df92a5cec5f7f6d279d861879933e90b350d6"
Sep 09 20:30:20.197135 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:20.197109    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg"]
Sep 09 20:30:20.202994 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:20.202969    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-21117b-29816430-tdqkg"]
Sep 09 20:30:21.009950 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:21.009911    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="effaa8aa-2715-4a27-889f-12f211980ed5" path="/var/lib/kubelet/pods/effaa8aa-2715-4a27-889f-12f211980ed5/volumes"
Sep 09 20:30:22.482748 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.482713    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"]
Sep 09 20:30:22.483119 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.483102    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effaa8aa-2715-4a27-889f-12f211980ed5" containerName="cleanup"
Sep 09 20:30:22.483119 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.483113    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="effaa8aa-2715-4a27-889f-12f211980ed5" containerName="cleanup"
Sep 09 20:30:22.483195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.483135    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effaa8aa-2715-4a27-889f-12f211980ed5" containerName="cleanup"
Sep 09 20:30:22.483195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.483142    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="effaa8aa-2715-4a27-889f-12f211980ed5" containerName="cleanup"
Sep 09 20:30:22.483254 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.483202    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="effaa8aa-2715-4a27-889f-12f211980ed5" containerName="cleanup"
Sep 09 20:30:22.483254 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.483210    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="effaa8aa-2715-4a27-889f-12f211980ed5" containerName="cleanup"
Sep 09 20:30:22.483254 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.483215    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="effaa8aa-2715-4a27-889f-12f211980ed5" containerName="cleanup"
Sep 09 20:30:22.486307 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.486291    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:30:22.488709 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.488677    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4defd2-dockercfg-sst6q\""
Sep 09 20:30:22.488839 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.488750    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4defd2\""
Sep 09 20:30:22.494570 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.494544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"]
Sep 09 20:30:22.574011 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.573984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vl25\" (UniqueName: \"kubernetes.io/projected/484ab58f-2800-4679-be53-38c319a335f5-kube-api-access-6vl25\") pod \"payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd\" (UID: \"484ab58f-2800-4679-be53-38c319a335f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:30:22.574167 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.574118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/484ab58f-2800-4679-be53-38c319a335f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd\" (UID: \"484ab58f-2800-4679-be53-38c319a335f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:30:22.675214 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.675170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/484ab58f-2800-4679-be53-38c319a335f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd\" (UID: \"484ab58f-2800-4679-be53-38c319a335f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:30:22.675405 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.675239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vl25\" (UniqueName: \"kubernetes.io/projected/484ab58f-2800-4679-be53-38c319a335f5-kube-api-access-6vl25\") pod \"payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd\" (UID: \"484ab58f-2800-4679-be53-38c319a335f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:30:22.675873 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.675852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/484ab58f-2800-4679-be53-38c319a335f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd\" (UID: \"484ab58f-2800-4679-be53-38c319a335f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:30:22.683522 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.683491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vl25\" (UniqueName: \"kubernetes.io/projected/484ab58f-2800-4679-be53-38c319a335f5-kube-api-access-6vl25\") pod \"payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd\" (UID: \"484ab58f-2800-4679-be53-38c319a335f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:30:22.797683 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:22.797597    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:30:23.070909 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:23.070885    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"]
Sep 09 20:30:23.072938 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:30:23.072906    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484ab58f_2800_4679_be53_38c319a335f5.slice/crio-48b3dd02f6145cb4a4d9ecf283d7834ae6eeab9d5db6538804ebc9a8c4345cc0 WatchSource:0}: Error finding container 48b3dd02f6145cb4a4d9ecf283d7834ae6eeab9d5db6538804ebc9a8c4345cc0: Status 404 returned error can't find the container with id 48b3dd02f6145cb4a4d9ecf283d7834ae6eeab9d5db6538804ebc9a8c4345cc0
Sep 09 20:30:23.183143 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:23.183081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd" event={"ID":"484ab58f-2800-4679-be53-38c319a335f5","Type":"ContainerStarted","Data":"95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e"}
Sep 09 20:30:23.183375 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:23.183191    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:30:23.183375 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:23.183220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd" event={"ID":"484ab58f-2800-4679-be53-38c319a335f5","Type":"ContainerStarted","Data":"48b3dd02f6145cb4a4d9ecf283d7834ae6eeab9d5db6538804ebc9a8c4345cc0"}
Sep 09 20:30:23.200094 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:23.200035    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd" podStartSLOduration=1.200016558 podStartE2EDuration="1.200016558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:23.19827313 +0000 UTC m=+844.691484115" watchObservedRunningTime="2026-09-09 20:30:23.200016558 +0000 UTC m=+844.693227548"
Sep 09 20:30:24.121211 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:24.121183    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:30:34.192076 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:34.192041    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:30:36.273042 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.273012    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"]
Sep 09 20:30:36.273599 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.273241    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d" podUID="4d4e0920-5b18-4a8b-bdeb-06d154ed6aca" containerName="payload-processing" containerID="cri-o://e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f" gracePeriod=30
Sep 09 20:30:36.281277 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.281247    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"]
Sep 09 20:30:36.281547 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.281504    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2" podUID="5f3164af-85f3-4dcb-ab78-a6722bc73b5b" containerName="payload-pre-processing" containerID="cri-o://c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3" gracePeriod=30
Sep 09 20:30:36.580534 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.580504    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:36.600619 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.600584    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:36.700379 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.700312    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-plugins-config-volume\") pod \"5f3164af-85f3-4dcb-ab78-a6722bc73b5b\" (UID: \"5f3164af-85f3-4dcb-ab78-a6722bc73b5b\") "
Sep 09 20:30:36.700574 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.700424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v77g7\" (UniqueName: \"kubernetes.io/projected/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-kube-api-access-v77g7\") pod \"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca\" (UID: \"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca\") "
Sep 09 20:30:36.700574 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.700464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gqkx\" (UniqueName: \"kubernetes.io/projected/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-kube-api-access-9gqkx\") pod \"5f3164af-85f3-4dcb-ab78-a6722bc73b5b\" (UID: \"5f3164af-85f3-4dcb-ab78-a6722bc73b5b\") "
Sep 09 20:30:36.700574 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.700500    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-plugins-config-volume\") pod \"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca\" (UID: \"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca\") "
Sep 09 20:30:36.700737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.700686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f3164af-85f3-4dcb-ab78-a6722bc73b5b" (UID: "5f3164af-85f3-4dcb-ab78-a6722bc73b5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:36.700904 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.700885    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:30:36.700949 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.700893    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d4e0920-5b18-4a8b-bdeb-06d154ed6aca" (UID: "4d4e0920-5b18-4a8b-bdeb-06d154ed6aca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:30:36.702723 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.702699    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-kube-api-access-v77g7" (OuterVolumeSpecName: "kube-api-access-v77g7") pod "4d4e0920-5b18-4a8b-bdeb-06d154ed6aca" (UID: "4d4e0920-5b18-4a8b-bdeb-06d154ed6aca"). InnerVolumeSpecName "kube-api-access-v77g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:36.702723 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.702705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-kube-api-access-9gqkx" (OuterVolumeSpecName: "kube-api-access-9gqkx") pod "5f3164af-85f3-4dcb-ab78-a6722bc73b5b" (UID: "5f3164af-85f3-4dcb-ab78-a6722bc73b5b"). InnerVolumeSpecName "kube-api-access-9gqkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:30:36.801777 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.801690    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v77g7\" (UniqueName: \"kubernetes.io/projected/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-kube-api-access-v77g7\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:30:36.801777 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.801720    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gqkx\" (UniqueName: \"kubernetes.io/projected/5f3164af-85f3-4dcb-ab78-a6722bc73b5b-kube-api-access-9gqkx\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:30:36.801777 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:36.801730    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:30:37.263521 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.263485    2581 generic.go:356] "Generic (PLEG): container finished" podID="5f3164af-85f3-4dcb-ab78-a6722bc73b5b" containerID="c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3" exitCode=0
Sep 09 20:30:37.263729 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.263539    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"
Sep 09 20:30:37.263729 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.263571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2" event={"ID":"5f3164af-85f3-4dcb-ab78-a6722bc73b5b","Type":"ContainerDied","Data":"c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3"}
Sep 09 20:30:37.263729 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.263634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2" event={"ID":"5f3164af-85f3-4dcb-ab78-a6722bc73b5b","Type":"ContainerDied","Data":"7f0a3c881acd7b1faa9ed4229cdea768344e5a407b96c28516690d446154b7e8"}
Sep 09 20:30:37.263729 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.263661    2581 scope.go:117] "RemoveContainer" containerID="c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3"
Sep 09 20:30:37.265115 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.265077    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d4e0920-5b18-4a8b-bdeb-06d154ed6aca" containerID="e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f" exitCode=0
Sep 09 20:30:37.265257 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.265141    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"
Sep 09 20:30:37.265257 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.265186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d" event={"ID":"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca","Type":"ContainerDied","Data":"e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f"}
Sep 09 20:30:37.265257 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.265225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d" event={"ID":"4d4e0920-5b18-4a8b-bdeb-06d154ed6aca","Type":"ContainerDied","Data":"7a869ff7fe0e75be49f2f74d95df6395ea22ae48839cc8704925f61568152056"}
Sep 09 20:30:37.272987 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.272968    2581 scope.go:117] "RemoveContainer" containerID="c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3"
Sep 09 20:30:37.273270 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:30:37.273252    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3\": container with ID starting with c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3 not found: ID does not exist" containerID="c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3"
Sep 09 20:30:37.273631 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.273278    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3"} err="failed to get container status \"c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3\": rpc error: code = NotFound desc = could not find container \"c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3\": container with ID starting with c71ea22a089617af06a720b93e704cde0c55f9f1150834a231a51ba5c54510e3 not found: ID does not exist"
Sep 09 20:30:37.273631 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.273294    2581 scope.go:117] "RemoveContainer" containerID="e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f"
Sep 09 20:30:37.281708 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.281691    2581 scope.go:117] "RemoveContainer" containerID="e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f"
Sep 09 20:30:37.281970 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:30:37.281949    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f\": container with ID starting with e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f not found: ID does not exist" containerID="e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f"
Sep 09 20:30:37.282033 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.281984    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f"} err="failed to get container status \"e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f\": rpc error: code = NotFound desc = could not find container \"e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f\": container with ID starting with e6af24a93ab5a9abaec8645cbbf456b43ee2aa45c4fbdf54ae0823fe1adf715f not found: ID does not exist"
Sep 09 20:30:37.286984 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.286958    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"]
Sep 09 20:30:37.296763 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.296739    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87ac1d4f-5f9db94cd-4cxg2"]
Sep 09 20:30:37.312751 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.312703    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"]
Sep 09 20:30:37.315061 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:37.315038    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87ac1d4f-76c5fff4f9-hjw2d"]
Sep 09 20:30:39.002315 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:39.002268    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d4e0920-5b18-4a8b-bdeb-06d154ed6aca" path="/var/lib/kubelet/pods/4d4e0920-5b18-4a8b-bdeb-06d154ed6aca/volumes"
Sep 09 20:30:39.002851 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:39.002799    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f3164af-85f3-4dcb-ab78-a6722bc73b5b" path="/var/lib/kubelet/pods/5f3164af-85f3-4dcb-ab78-a6722bc73b5b/volumes"
Sep 09 20:30:43.710827 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.710793    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"]
Sep 09 20:30:43.711208 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.711146    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f3164af-85f3-4dcb-ab78-a6722bc73b5b" containerName="payload-pre-processing"
Sep 09 20:30:43.711208 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.711157    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f3164af-85f3-4dcb-ab78-a6722bc73b5b" containerName="payload-pre-processing"
Sep 09 20:30:43.711208 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.711179    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4e0920-5b18-4a8b-bdeb-06d154ed6aca" containerName="payload-processing"
Sep 09 20:30:43.711208 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.711185    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4e0920-5b18-4a8b-bdeb-06d154ed6aca" containerName="payload-processing"
Sep 09 20:30:43.711208 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.711198    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="effaa8aa-2715-4a27-889f-12f211980ed5" containerName="cleanup"
Sep 09 20:30:43.711208 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.711204    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="effaa8aa-2715-4a27-889f-12f211980ed5" containerName="cleanup"
Sep 09 20:30:43.711428 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.711256    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4e0920-5b18-4a8b-bdeb-06d154ed6aca" containerName="payload-processing"
Sep 09 20:30:43.711428 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.711264    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f3164af-85f3-4dcb-ab78-a6722bc73b5b" containerName="payload-pre-processing"
Sep 09 20:30:43.714189 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.714171    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:30:43.716690 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.716670    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-25e8b2a0\""
Sep 09 20:30:43.716817 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.716726    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-25e8b2a0-dockercfg-dszdb\""
Sep 09 20:30:43.724785 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.724764    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"]
Sep 09 20:30:43.872973 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.872939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b65a83c-2935-47b4-b18d-1ee545c33625-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4\" (UID: \"1b65a83c-2935-47b4-b18d-1ee545c33625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:30:43.873162 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.873029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8zrl\" (UniqueName: \"kubernetes.io/projected/1b65a83c-2935-47b4-b18d-1ee545c33625-kube-api-access-b8zrl\") pod \"payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4\" (UID: \"1b65a83c-2935-47b4-b18d-1ee545c33625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:30:43.973599 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.973510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8zrl\" (UniqueName: \"kubernetes.io/projected/1b65a83c-2935-47b4-b18d-1ee545c33625-kube-api-access-b8zrl\") pod \"payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4\" (UID: \"1b65a83c-2935-47b4-b18d-1ee545c33625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:30:43.973599 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.973589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b65a83c-2935-47b4-b18d-1ee545c33625-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4\" (UID: \"1b65a83c-2935-47b4-b18d-1ee545c33625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:30:43.974199 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.974181    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b65a83c-2935-47b4-b18d-1ee545c33625-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4\" (UID: \"1b65a83c-2935-47b4-b18d-1ee545c33625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:30:43.981671 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:43.981644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8zrl\" (UniqueName: \"kubernetes.io/projected/1b65a83c-2935-47b4-b18d-1ee545c33625-kube-api-access-b8zrl\") pod \"payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4\" (UID: \"1b65a83c-2935-47b4-b18d-1ee545c33625\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:30:44.024814 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:44.024775    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:30:44.297254 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:44.297221    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"]
Sep 09 20:30:44.299231 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:30:44.299203    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b65a83c_2935_47b4_b18d_1ee545c33625.slice/crio-de972a35c135e4ca6daff7b574ecdd1b9579dc3af6f1b9e7cf9c0c2b08acfc2a WatchSource:0}: Error finding container de972a35c135e4ca6daff7b574ecdd1b9579dc3af6f1b9e7cf9c0c2b08acfc2a: Status 404 returned error can't find the container with id de972a35c135e4ca6daff7b574ecdd1b9579dc3af6f1b9e7cf9c0c2b08acfc2a
Sep 09 20:30:44.315068 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:44.314948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4" event={"ID":"1b65a83c-2935-47b4-b18d-1ee545c33625","Type":"ContainerStarted","Data":"de972a35c135e4ca6daff7b574ecdd1b9579dc3af6f1b9e7cf9c0c2b08acfc2a"}
Sep 09 20:30:45.323157 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:45.323123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4" event={"ID":"1b65a83c-2935-47b4-b18d-1ee545c33625","Type":"ContainerStarted","Data":"f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d"}
Sep 09 20:30:45.323613 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:45.323258    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:30:45.339742 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:45.339688    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4" podStartSLOduration=2.339672233 podStartE2EDuration="2.339672233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:30:45.337937585 +0000 UTC m=+866.831148570" watchObservedRunningTime="2026-09-09 20:30:45.339672233 +0000 UTC m=+866.832883254"
Sep 09 20:30:56.331182 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:30:56.331149    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:31:05.500035 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:05.499998    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"]
Sep 09 20:31:05.504899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:05.504882    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"
Sep 09 20:31:05.507329 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:05.507303    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:31:05.507618 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:05.507335    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:31:05.509377 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:05.509336    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"]
Sep 09 20:31:05.577322 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:05.577289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ce8bcd-853b-42d2-bd05-451a9eefbcfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8\" (UID: \"c6ce8bcd-853b-42d2-bd05-451a9eefbcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"
Sep 09 20:31:05.678508 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:05.678468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ce8bcd-853b-42d2-bd05-451a9eefbcfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8\" (UID: \"c6ce8bcd-853b-42d2-bd05-451a9eefbcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"
Sep 09 20:31:05.679046 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:05.679026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ce8bcd-853b-42d2-bd05-451a9eefbcfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8\" (UID: \"c6ce8bcd-853b-42d2-bd05-451a9eefbcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"
Sep 09 20:31:05.835247 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:05.835162    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"
Sep 09 20:31:06.109000 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:06.108973    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"]
Sep 09 20:31:06.110842 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:31:06.110808    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ce8bcd_853b_42d2_bd05_451a9eefbcfc.slice/crio-73e7132b2ffce8e9151521086a5ccf9c9177f0a524a1b87030c1b092ef6986af WatchSource:0}: Error finding container 73e7132b2ffce8e9151521086a5ccf9c9177f0a524a1b87030c1b092ef6986af: Status 404 returned error can't find the container with id 73e7132b2ffce8e9151521086a5ccf9c9177f0a524a1b87030c1b092ef6986af
Sep 09 20:31:06.444528 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:06.444499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8" event={"ID":"c6ce8bcd-853b-42d2-bd05-451a9eefbcfc","Type":"ContainerStarted","Data":"73e7132b2ffce8e9151521086a5ccf9c9177f0a524a1b87030c1b092ef6986af"}
Sep 09 20:31:07.452627 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:07.452589    2581 generic.go:356] "Generic (PLEG): container finished" podID="c6ce8bcd-853b-42d2-bd05-451a9eefbcfc" containerID="9421c8b52ffe0425b443e3a9d6b259910c8413445e95366c1c02d97aacc2da7d" exitCode=0
Sep 09 20:31:07.453101 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:07.452639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8" event={"ID":"c6ce8bcd-853b-42d2-bd05-451a9eefbcfc","Type":"ContainerDied","Data":"9421c8b52ffe0425b443e3a9d6b259910c8413445e95366c1c02d97aacc2da7d"}
Sep 09 20:31:08.599464 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:08.599443    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"
Sep 09 20:31:08.705483 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:08.705457    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ce8bcd-853b-42d2-bd05-451a9eefbcfc-maas-api-service-ca\") pod \"c6ce8bcd-853b-42d2-bd05-451a9eefbcfc\" (UID: \"c6ce8bcd-853b-42d2-bd05-451a9eefbcfc\") "
Sep 09 20:31:08.705792 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:08.705770    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ce8bcd-853b-42d2-bd05-451a9eefbcfc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6ce8bcd-853b-42d2-bd05-451a9eefbcfc" (UID: "c6ce8bcd-853b-42d2-bd05-451a9eefbcfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:08.806594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:08.806517    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6ce8bcd-853b-42d2-bd05-451a9eefbcfc-maas-api-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:31:09.464328 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:09.464297    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"
Sep 09 20:31:09.464328 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:09.464323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8" event={"ID":"c6ce8bcd-853b-42d2-bd05-451a9eefbcfc","Type":"ContainerDied","Data":"73e7132b2ffce8e9151521086a5ccf9c9177f0a524a1b87030c1b092ef6986af"}
Sep 09 20:31:09.464566 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:09.464391    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73e7132b2ffce8e9151521086a5ccf9c9177f0a524a1b87030c1b092ef6986af"
Sep 09 20:31:12.055265 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.055235    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"]
Sep 09 20:31:12.055679 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.055663    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ce8bcd-853b-42d2-bd05-451a9eefbcfc" containerName="revoke-keys"
Sep 09 20:31:12.055728 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.055683    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ce8bcd-853b-42d2-bd05-451a9eefbcfc" containerName="revoke-keys"
Sep 09 20:31:12.055763 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.055757    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ce8bcd-853b-42d2-bd05-451a9eefbcfc" containerName="revoke-keys"
Sep 09 20:31:12.058840 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.058825    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.062182 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.062159    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-812760\"/\"autores-model-812760-kserve-self-signed-certs\""
Sep 09 20:31:12.062182 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.062171    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-812760\"/\"openshift-service-ca.crt\""
Sep 09 20:31:12.062417 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.062193    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-812760\"/\"default-dockercfg-5kq2z\""
Sep 09 20:31:12.062417 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.062162    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-812760\"/\"kube-root-ca.crt\""
Sep 09 20:31:12.068510 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.068488    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"]
Sep 09 20:31:12.138007 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.137957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tls-certs\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.138007 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.138005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwlg\" (UniqueName: \"kubernetes.io/projected/4ea25009-4a2e-4c3b-ad08-eef3d899e272-kube-api-access-dzwlg\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.138271 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.138043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-home\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.138271 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.138096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tmp-dir\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.138439 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.138309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-dshm\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.138439 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.138398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-model-cache\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.185490 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.185458    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"]
Sep 09 20:31:12.185747 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.185701    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4" podUID="1b65a83c-2935-47b4-b18d-1ee545c33625" containerName="payload-pre-processing" containerID="cri-o://f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d" gracePeriod=30
Sep 09 20:31:12.239944 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.239911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-model-cache\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.240097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.239990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tls-certs\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.240097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.240007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwlg\" (UniqueName: \"kubernetes.io/projected/4ea25009-4a2e-4c3b-ad08-eef3d899e272-kube-api-access-dzwlg\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.240097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.240029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-home\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.240097 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.240047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tmp-dir\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.240321 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.240101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-dshm\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.240404 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.240383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-model-cache\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.240464 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.240403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tmp-dir\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.240523 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.240465    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-home\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.242400 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.242377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-dshm\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.242716 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.242700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tls-certs\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.247442 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.247419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwlg\" (UniqueName: \"kubernetes.io/projected/4ea25009-4a2e-4c3b-ad08-eef3d899e272-kube-api-access-dzwlg\") pod \"autores-model-812760-kserve-744c7d6ff9-hs9xf\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") " pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.340365 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.340263    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"]
Sep 09 20:31:12.344266 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.344241    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-25e8b2a0-2hdk8"]
Sep 09 20:31:12.369711 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.369682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:12.456445 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.456415    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:31:12.487054 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.487022    2581 generic.go:356] "Generic (PLEG): container finished" podID="1b65a83c-2935-47b4-b18d-1ee545c33625" containerID="f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d" exitCode=0
Sep 09 20:31:12.487230 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.487078    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"
Sep 09 20:31:12.487230 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.487070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4" event={"ID":"1b65a83c-2935-47b4-b18d-1ee545c33625","Type":"ContainerDied","Data":"f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d"}
Sep 09 20:31:12.487230 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.487169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4" event={"ID":"1b65a83c-2935-47b4-b18d-1ee545c33625","Type":"ContainerDied","Data":"de972a35c135e4ca6daff7b574ecdd1b9579dc3af6f1b9e7cf9c0c2b08acfc2a"}
Sep 09 20:31:12.487230 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.487200    2581 scope.go:117] "RemoveContainer" containerID="f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d"
Sep 09 20:31:12.497537 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.497506    2581 scope.go:117] "RemoveContainer" containerID="f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d"
Sep 09 20:31:12.497847 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:31:12.497827    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d\": container with ID starting with f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d not found: ID does not exist" containerID="f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d"
Sep 09 20:31:12.497930 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.497857    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d"} err="failed to get container status \"f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d\": rpc error: code = NotFound desc = could not find container \"f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d\": container with ID starting with f8f8da159b769f1011f57ce015bb3dcbd5b46f9317fc47df90fec3bc05da4f1d not found: ID does not exist"
Sep 09 20:31:12.542549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.542503    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8zrl\" (UniqueName: \"kubernetes.io/projected/1b65a83c-2935-47b4-b18d-1ee545c33625-kube-api-access-b8zrl\") pod \"1b65a83c-2935-47b4-b18d-1ee545c33625\" (UID: \"1b65a83c-2935-47b4-b18d-1ee545c33625\") "
Sep 09 20:31:12.542769 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.542587    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b65a83c-2935-47b4-b18d-1ee545c33625-plugins-config-volume\") pod \"1b65a83c-2935-47b4-b18d-1ee545c33625\" (UID: \"1b65a83c-2935-47b4-b18d-1ee545c33625\") "
Sep 09 20:31:12.542999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.542971    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b65a83c-2935-47b4-b18d-1ee545c33625-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1b65a83c-2935-47b4-b18d-1ee545c33625" (UID: "1b65a83c-2935-47b4-b18d-1ee545c33625"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:12.544724 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.544702    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b65a83c-2935-47b4-b18d-1ee545c33625-kube-api-access-b8zrl" (OuterVolumeSpecName: "kube-api-access-b8zrl") pod "1b65a83c-2935-47b4-b18d-1ee545c33625" (UID: "1b65a83c-2935-47b4-b18d-1ee545c33625"). InnerVolumeSpecName "kube-api-access-b8zrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:12.644334 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.644240    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8zrl\" (UniqueName: \"kubernetes.io/projected/1b65a83c-2935-47b4-b18d-1ee545c33625-kube-api-access-b8zrl\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:31:12.644334 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.644283    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b65a83c-2935-47b4-b18d-1ee545c33625-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:31:12.647447 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.647423    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"]
Sep 09 20:31:12.648733 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:31:12.648702    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea25009_4a2e_4c3b_ad08_eef3d899e272.slice/crio-5cea6cc3a6fdec6e8b8a8d9909f790e3eed6b618a503321e6443db5059d8c322 WatchSource:0}: Error finding container 5cea6cc3a6fdec6e8b8a8d9909f790e3eed6b618a503321e6443db5059d8c322: Status 404 returned error can't find the container with id 5cea6cc3a6fdec6e8b8a8d9909f790e3eed6b618a503321e6443db5059d8c322
Sep 09 20:31:12.813955 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.813921    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"]
Sep 09 20:31:12.817332 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:12.817305    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-25e8b2a0-557b5446c8-sn9q4"]
Sep 09 20:31:13.002863 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:13.002829    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b65a83c-2935-47b4-b18d-1ee545c33625" path="/var/lib/kubelet/pods/1b65a83c-2935-47b4-b18d-1ee545c33625/volumes"
Sep 09 20:31:13.003214 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:13.003201    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6ce8bcd-853b-42d2-bd05-451a9eefbcfc" path="/var/lib/kubelet/pods/c6ce8bcd-853b-42d2-bd05-451a9eefbcfc/volumes"
Sep 09 20:31:13.496306 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:13.496272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf" event={"ID":"4ea25009-4a2e-4c3b-ad08-eef3d899e272","Type":"ContainerStarted","Data":"16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea"}
Sep 09 20:31:13.496812 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:13.496323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf" event={"ID":"4ea25009-4a2e-4c3b-ad08-eef3d899e272","Type":"ContainerStarted","Data":"5cea6cc3a6fdec6e8b8a8d9909f790e3eed6b618a503321e6443db5059d8c322"}
Sep 09 20:31:13.513916 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:13.513867    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf" podStartSLOduration=1.513851968 podStartE2EDuration="1.513851968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:13.511989209 +0000 UTC m=+895.005200192" watchObservedRunningTime="2026-09-09 20:31:13.513851968 +0000 UTC m=+895.007062995"
Sep 09 20:31:18.985439 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:18.985411    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:31:18.985899 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:18.985461    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:31:20.154957 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.154926    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"]
Sep 09 20:31:20.155320 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.155296    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b65a83c-2935-47b4-b18d-1ee545c33625" containerName="payload-pre-processing"
Sep 09 20:31:20.155320 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.155307    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b65a83c-2935-47b4-b18d-1ee545c33625" containerName="payload-pre-processing"
Sep 09 20:31:20.155452 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.155389    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b65a83c-2935-47b4-b18d-1ee545c33625" containerName="payload-pre-processing"
Sep 09 20:31:20.159881 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.159865    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:20.162266 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.162242    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4275e4e8-dockercfg-9vxcf\""
Sep 09 20:31:20.162411 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.162242    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4275e4e8\""
Sep 09 20:31:20.169245 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.169219    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"]
Sep 09 20:31:20.312533 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.312496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80127ef6-0063-40cf-bd54-46a4581d489d-plugins-config-volume\") pod \"payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr\" (UID: \"80127ef6-0063-40cf-bd54-46a4581d489d\") " pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:20.312730 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.312562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hws4\" (UniqueName: \"kubernetes.io/projected/80127ef6-0063-40cf-bd54-46a4581d489d-kube-api-access-4hws4\") pod \"payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr\" (UID: \"80127ef6-0063-40cf-bd54-46a4581d489d\") " pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:20.413312 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.413223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hws4\" (UniqueName: \"kubernetes.io/projected/80127ef6-0063-40cf-bd54-46a4581d489d-kube-api-access-4hws4\") pod \"payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr\" (UID: \"80127ef6-0063-40cf-bd54-46a4581d489d\") " pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:20.413497 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.413392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80127ef6-0063-40cf-bd54-46a4581d489d-plugins-config-volume\") pod \"payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr\" (UID: \"80127ef6-0063-40cf-bd54-46a4581d489d\") " pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:20.414063 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.414035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80127ef6-0063-40cf-bd54-46a4581d489d-plugins-config-volume\") pod \"payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr\" (UID: \"80127ef6-0063-40cf-bd54-46a4581d489d\") " pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:20.421432 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.421409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hws4\" (UniqueName: \"kubernetes.io/projected/80127ef6-0063-40cf-bd54-46a4581d489d-kube-api-access-4hws4\") pod \"payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr\" (UID: \"80127ef6-0063-40cf-bd54-46a4581d489d\") " pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:20.470591 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.470566    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:20.750758 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:20.750719    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"]
Sep 09 20:31:20.753142 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:31:20.753113    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80127ef6_0063_40cf_bd54_46a4581d489d.slice/crio-7e36b47a47791c0f5d6cc85b94cb7172208dcf93180f9e0fa312906257277450 WatchSource:0}: Error finding container 7e36b47a47791c0f5d6cc85b94cb7172208dcf93180f9e0fa312906257277450: Status 404 returned error can't find the container with id 7e36b47a47791c0f5d6cc85b94cb7172208dcf93180f9e0fa312906257277450
Sep 09 20:31:21.550190 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:21.550145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr" event={"ID":"80127ef6-0063-40cf-bd54-46a4581d489d","Type":"ContainerStarted","Data":"d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1"}
Sep 09 20:31:21.550612 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:21.550215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr" event={"ID":"80127ef6-0063-40cf-bd54-46a4581d489d","Type":"ContainerStarted","Data":"7e36b47a47791c0f5d6cc85b94cb7172208dcf93180f9e0fa312906257277450"}
Sep 09 20:31:21.550612 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:21.550246    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:21.569224 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:21.569162    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr" podStartSLOduration=1.569145282 podStartE2EDuration="1.569145282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:21.56733178 +0000 UTC m=+903.060542778" watchObservedRunningTime="2026-09-09 20:31:21.569145282 +0000 UTC m=+903.062356265"
Sep 09 20:31:22.370690 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:22.370651    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:22.370690 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:22.370699    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:22.384361 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:22.384323    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:22.567738 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:22.567703    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:31:32.558402 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:32.558373    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:47.747694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:47.747658    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"]
Sep 09 20:31:47.751303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:47.751283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"
Sep 09 20:31:47.753794 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:47.753776    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:31:47.753880 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:47.753776    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:31:47.758961 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:47.758930    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"]
Sep 09 20:31:47.863078 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:47.863036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc6fcd8-ffe2-4bf9-bd57-8b86b509589b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-812760-4c2zt\" (UID: \"edc6fcd8-ffe2-4bf9-bd57-8b86b509589b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"
Sep 09 20:31:47.964128 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:47.964089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc6fcd8-ffe2-4bf9-bd57-8b86b509589b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-812760-4c2zt\" (UID: \"edc6fcd8-ffe2-4bf9-bd57-8b86b509589b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"
Sep 09 20:31:47.964736 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:47.964717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc6fcd8-ffe2-4bf9-bd57-8b86b509589b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-812760-4c2zt\" (UID: \"edc6fcd8-ffe2-4bf9-bd57-8b86b509589b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"
Sep 09 20:31:48.064422 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:48.064311    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"
Sep 09 20:31:48.342532 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:48.342504    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"]
Sep 09 20:31:48.345414 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:31:48.345384    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc6fcd8_ffe2_4bf9_bd57_8b86b509589b.slice/crio-f275d6ac860dc74ea8d7eadaa47323abc20b3c244f53fae4708ec359ec56b9df WatchSource:0}: Error finding container f275d6ac860dc74ea8d7eadaa47323abc20b3c244f53fae4708ec359ec56b9df: Status 404 returned error can't find the container with id f275d6ac860dc74ea8d7eadaa47323abc20b3c244f53fae4708ec359ec56b9df
Sep 09 20:31:48.703534 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:48.703497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt" event={"ID":"edc6fcd8-ffe2-4bf9-bd57-8b86b509589b","Type":"ContainerStarted","Data":"f990ea45ef085ecd6c5cae78cc52f0158643bcc1cf3122e3bd8e684fdaf3f53f"}
Sep 09 20:31:48.703739 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:48.703552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt" event={"ID":"edc6fcd8-ffe2-4bf9-bd57-8b86b509589b","Type":"ContainerStarted","Data":"f275d6ac860dc74ea8d7eadaa47323abc20b3c244f53fae4708ec359ec56b9df"}
Sep 09 20:31:48.719598 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:48.719548    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt" podStartSLOduration=1.719533895 podStartE2EDuration="1.719533895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:48.716968359 +0000 UTC m=+930.210179343" watchObservedRunningTime="2026-09-09 20:31:48.719533895 +0000 UTC m=+930.212744908"
Sep 09 20:31:49.711268 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:49.711229    2581 generic.go:356] "Generic (PLEG): container finished" podID="edc6fcd8-ffe2-4bf9-bd57-8b86b509589b" containerID="f990ea45ef085ecd6c5cae78cc52f0158643bcc1cf3122e3bd8e684fdaf3f53f" exitCode=0
Sep 09 20:31:49.711691 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:49.711293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt" event={"ID":"edc6fcd8-ffe2-4bf9-bd57-8b86b509589b","Type":"ContainerDied","Data":"f990ea45ef085ecd6c5cae78cc52f0158643bcc1cf3122e3bd8e684fdaf3f53f"}
Sep 09 20:31:50.860305 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:50.860277    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"
Sep 09 20:31:50.890643 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:50.890612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc6fcd8-ffe2-4bf9-bd57-8b86b509589b-maas-api-service-ca\") pod \"edc6fcd8-ffe2-4bf9-bd57-8b86b509589b\" (UID: \"edc6fcd8-ffe2-4bf9-bd57-8b86b509589b\") "
Sep 09 20:31:50.890974 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:50.890949    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc6fcd8-ffe2-4bf9-bd57-8b86b509589b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edc6fcd8-ffe2-4bf9-bd57-8b86b509589b" (UID: "edc6fcd8-ffe2-4bf9-bd57-8b86b509589b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:50.991454 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:50.991370    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc6fcd8-ffe2-4bf9-bd57-8b86b509589b-maas-api-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:31:51.723918 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:51.723892    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"
Sep 09 20:31:51.723918 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:51.723896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt" event={"ID":"edc6fcd8-ffe2-4bf9-bd57-8b86b509589b","Type":"ContainerDied","Data":"f275d6ac860dc74ea8d7eadaa47323abc20b3c244f53fae4708ec359ec56b9df"}
Sep 09 20:31:51.724145 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:51.723947    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f275d6ac860dc74ea8d7eadaa47323abc20b3c244f53fae4708ec359ec56b9df"
Sep 09 20:31:52.752809 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.752766    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"]
Sep 09 20:31:52.755314 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.753400    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc6fcd8-ffe2-4bf9-bd57-8b86b509589b" containerName="revoke-keys"
Sep 09 20:31:52.755314 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.753418    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc6fcd8-ffe2-4bf9-bd57-8b86b509589b" containerName="revoke-keys"
Sep 09 20:31:52.755314 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.753544    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc6fcd8-ffe2-4bf9-bd57-8b86b509589b" containerName="revoke-keys"
Sep 09 20:31:52.765403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.765375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.767098 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.767074    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"]
Sep 09 20:31:52.769712 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.769693    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-21117b\"/\"kube-root-ca.crt\""
Sep 09 20:31:52.770600 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.770576    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-21117b\"/\"test-model-21117b-kserve-self-signed-certs\""
Sep 09 20:31:52.770843 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.770823    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-21117b\"/\"default-dockercfg-ghzxc\""
Sep 09 20:31:52.770967 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.770864    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-21117b\"/\"openshift-service-ca.crt\""
Sep 09 20:31:52.809175 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.809145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496a7626-e8a3-4ade-9138-ae2123dfd840-tls-certs\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.809377 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.809182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-model-cache\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.809377 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.809211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65hgp\" (UniqueName: \"kubernetes.io/projected/496a7626-e8a3-4ade-9138-ae2123dfd840-kube-api-access-65hgp\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.809377 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.809290    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-dshm\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.809377 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.809365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-home\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.809595 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.809484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-tmp-dir\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.909974 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.909934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-tmp-dir\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.910159 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.909982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496a7626-e8a3-4ade-9138-ae2123dfd840-tls-certs\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.910159 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.910104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-model-cache\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.910277 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.910167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65hgp\" (UniqueName: \"kubernetes.io/projected/496a7626-e8a3-4ade-9138-ae2123dfd840-kube-api-access-65hgp\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.910277 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.910199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-dshm\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.910277 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.910242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-home\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.910481 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.910432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-tmp-dir\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.910538 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.910479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-model-cache\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.910577 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.910568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-home\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.912582 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.912558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-dshm\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.912792 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.912773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496a7626-e8a3-4ade-9138-ae2123dfd840-tls-certs\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:52.918814 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:52.918793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65hgp\" (UniqueName: \"kubernetes.io/projected/496a7626-e8a3-4ade-9138-ae2123dfd840-kube-api-access-65hgp\") pod \"test-model-21117b-kserve-67c54df97c-tk5sb\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") " pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:53.074634 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:53.074543    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:31:53.354323 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:53.354288    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"]
Sep 09 20:31:53.356245 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:31:53.356221    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod496a7626_e8a3_4ade_9138_ae2123dfd840.slice/crio-a19734e16de68bb23c0595f199ed726f61d23fc5afbb1bde7bc10530645102fd WatchSource:0}: Error finding container a19734e16de68bb23c0595f199ed726f61d23fc5afbb1bde7bc10530645102fd: Status 404 returned error can't find the container with id a19734e16de68bb23c0595f199ed726f61d23fc5afbb1bde7bc10530645102fd
Sep 09 20:31:53.737987 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:53.737945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb" event={"ID":"496a7626-e8a3-4ade-9138-ae2123dfd840","Type":"ContainerStarted","Data":"d6818a4104054e7603b13a2aef050b2ac3a8b3a0737616c10be698cdac90594d"}
Sep 09 20:31:53.737987 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:53.737991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb" event={"ID":"496a7626-e8a3-4ade-9138-ae2123dfd840","Type":"ContainerStarted","Data":"a19734e16de68bb23c0595f199ed726f61d23fc5afbb1bde7bc10530645102fd"}
Sep 09 20:31:53.762203 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:53.762156    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb" podStartSLOduration=1.762139709 podStartE2EDuration="1.762139709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:53.759614834 +0000 UTC m=+935.252825830" watchObservedRunningTime="2026-09-09 20:31:53.762139709 +0000 UTC m=+935.255350690"
Sep 09 20:31:58.159612 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.159571    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"]
Sep 09 20:31:58.160062 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.159832    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw" podUID="dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156" containerName="payload-processing" containerID="cri-o://891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984" gracePeriod=30
Sep 09 20:31:58.376084 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.376047    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"]
Sep 09 20:31:58.376739 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.376703    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr" podUID="80127ef6-0063-40cf-bd54-46a4581d489d" containerName="payload-processing" containerID="cri-o://d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1" gracePeriod=30
Sep 09 20:31:58.440374 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.440315    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"]
Sep 09 20:31:58.446547 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.446518    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-812760-4c2zt"]
Sep 09 20:31:58.506645 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.506615    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:31:58.559741 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.559715    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkmr5\" (UniqueName: \"kubernetes.io/projected/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-kube-api-access-dkmr5\") pod \"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156\" (UID: \"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156\") "
Sep 09 20:31:58.559848 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.559759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-plugins-config-volume\") pod \"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156\" (UID: \"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156\") "
Sep 09 20:31:58.560158 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.560130    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156" (UID: "dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:58.562018 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.561994    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-kube-api-access-dkmr5" (OuterVolumeSpecName: "kube-api-access-dkmr5") pod "dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156" (UID: "dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156"). InnerVolumeSpecName "kube-api-access-dkmr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:58.642508 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.642474    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:58.660791 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.660754    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkmr5\" (UniqueName: \"kubernetes.io/projected/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-kube-api-access-dkmr5\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:31:58.660791 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.660786    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:31:58.762155 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.762118    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80127ef6-0063-40cf-bd54-46a4581d489d-plugins-config-volume\") pod \"80127ef6-0063-40cf-bd54-46a4581d489d\" (UID: \"80127ef6-0063-40cf-bd54-46a4581d489d\") "
Sep 09 20:31:58.762386 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.762193    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hws4\" (UniqueName: \"kubernetes.io/projected/80127ef6-0063-40cf-bd54-46a4581d489d-kube-api-access-4hws4\") pod \"80127ef6-0063-40cf-bd54-46a4581d489d\" (UID: \"80127ef6-0063-40cf-bd54-46a4581d489d\") "
Sep 09 20:31:58.762578 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.762549    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80127ef6-0063-40cf-bd54-46a4581d489d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80127ef6-0063-40cf-bd54-46a4581d489d" (UID: "80127ef6-0063-40cf-bd54-46a4581d489d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:31:58.764572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.764538    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80127ef6-0063-40cf-bd54-46a4581d489d-kube-api-access-4hws4" (OuterVolumeSpecName: "kube-api-access-4hws4") pod "80127ef6-0063-40cf-bd54-46a4581d489d" (UID: "80127ef6-0063-40cf-bd54-46a4581d489d"). InnerVolumeSpecName "kube-api-access-4hws4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:58.768651 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.768621    2581 generic.go:356] "Generic (PLEG): container finished" podID="dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156" containerID="891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984" exitCode=0
Sep 09 20:31:58.768799 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.768688    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"
Sep 09 20:31:58.768799 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.768707    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw" event={"ID":"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156","Type":"ContainerDied","Data":"891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984"}
Sep 09 20:31:58.768799 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.768795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw" event={"ID":"dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156","Type":"ContainerDied","Data":"e4fd213cb7d6550eaed07d3b8eb13e8af7bca881db0f09c997e12af52706e6b1"}
Sep 09 20:31:58.768936 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.768824    2581 scope.go:117] "RemoveContainer" containerID="891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984"
Sep 09 20:31:58.770290 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.770265    2581 generic.go:356] "Generic (PLEG): container finished" podID="80127ef6-0063-40cf-bd54-46a4581d489d" containerID="d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1" exitCode=0
Sep 09 20:31:58.770437 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.770300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr" event={"ID":"80127ef6-0063-40cf-bd54-46a4581d489d","Type":"ContainerDied","Data":"d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1"}
Sep 09 20:31:58.770437 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.770323    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"
Sep 09 20:31:58.770437 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.770387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr" event={"ID":"80127ef6-0063-40cf-bd54-46a4581d489d","Type":"ContainerDied","Data":"7e36b47a47791c0f5d6cc85b94cb7172208dcf93180f9e0fa312906257277450"}
Sep 09 20:31:58.780170 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.780145    2581 scope.go:117] "RemoveContainer" containerID="891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984"
Sep 09 20:31:58.780530 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:31:58.780502    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984\": container with ID starting with 891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984 not found: ID does not exist" containerID="891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984"
Sep 09 20:31:58.780624 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.780538    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984"} err="failed to get container status \"891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984\": rpc error: code = NotFound desc = could not find container \"891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984\": container with ID starting with 891070d9d543250fc688d10839b75e5e796671d4a1398055e70ea402b11f6984 not found: ID does not exist"
Sep 09 20:31:58.780624 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.780556    2581 scope.go:117] "RemoveContainer" containerID="d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1"
Sep 09 20:31:58.790662 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.790630    2581 scope.go:117] "RemoveContainer" containerID="d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1"
Sep 09 20:31:58.792880 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:31:58.792847    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1\": container with ID starting with d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1 not found: ID does not exist" containerID="d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1"
Sep 09 20:31:58.793025 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.792886    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1"} err="failed to get container status \"d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1\": rpc error: code = NotFound desc = could not find container \"d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1\": container with ID starting with d11887de1e212b39b8f6c81dea3eee150598435340c2dd2e737e161fca05a2e1 not found: ID does not exist"
Sep 09 20:31:58.833690 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.833655    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"]
Sep 09 20:31:58.848349 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.848300    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-812760-9d7fb49cc-cpwvw"]
Sep 09 20:31:58.863596 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.863564    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hws4\" (UniqueName: \"kubernetes.io/projected/80127ef6-0063-40cf-bd54-46a4581d489d-kube-api-access-4hws4\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:31:58.863752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.863600    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80127ef6-0063-40cf-bd54-46a4581d489d-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:31:58.886461 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.886424    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"]
Sep 09 20:31:58.899043 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:58.898997    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4275e4e8-547cb69fc5-wwdxr"]
Sep 09 20:31:59.004556 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:59.004521    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80127ef6-0063-40cf-bd54-46a4581d489d" path="/var/lib/kubelet/pods/80127ef6-0063-40cf-bd54-46a4581d489d/volumes"
Sep 09 20:31:59.004886 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:59.004871    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156" path="/var/lib/kubelet/pods/dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156/volumes"
Sep 09 20:31:59.005181 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:31:59.005169    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc6fcd8-ffe2-4bf9-bd57-8b86b509589b" path="/var/lib/kubelet/pods/edc6fcd8-ffe2-4bf9-bd57-8b86b509589b/volumes"
Sep 09 20:32:03.075573 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:03.075526    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:32:03.075573 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:03.075579    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:32:03.088365 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:03.088309    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:32:03.807658 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:03.807628    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"]
Sep 09 20:32:03.814618 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:03.814591    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:32:16.529704 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.529671    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"]
Sep 09 20:32:16.530101 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.530048    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80127ef6-0063-40cf-bd54-46a4581d489d" containerName="payload-processing"
Sep 09 20:32:16.530101 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.530059    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="80127ef6-0063-40cf-bd54-46a4581d489d" containerName="payload-processing"
Sep 09 20:32:16.530101 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.530079    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156" containerName="payload-processing"
Sep 09 20:32:16.530101 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.530084    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156" containerName="payload-processing"
Sep 09 20:32:16.530236 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.530150    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6931e6-f18f-45c4-a2fd-bdd3a7ecb156" containerName="payload-processing"
Sep 09 20:32:16.530236 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.530162    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="80127ef6-0063-40cf-bd54-46a4581d489d" containerName="payload-processing"
Sep 09 20:32:16.533001 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.532985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.535508 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.535486    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-836c5d71-kserve-self-signed-certs\""
Sep 09 20:32:16.542126 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.542100    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"]
Sep 09 20:32:16.635983 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.635943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-dshm\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.636165 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.635990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-tmp-dir\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.636165 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.636073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjzsh\" (UniqueName: \"kubernetes.io/projected/7537e768-3b42-436a-bc1b-a1d067fd30b7-kube-api-access-mjzsh\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.636165 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.636117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-home\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.636276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.636182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-model-cache\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.636276 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.636203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7537e768-3b42-436a-bc1b-a1d067fd30b7-tls-certs\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.737680 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.737645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjzsh\" (UniqueName: \"kubernetes.io/projected/7537e768-3b42-436a-bc1b-a1d067fd30b7-kube-api-access-mjzsh\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.737680 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.737679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-home\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.737943 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.737720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-model-cache\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.737943 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.737740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7537e768-3b42-436a-bc1b-a1d067fd30b7-tls-certs\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.737943 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.737866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-dshm\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.737943 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.737927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-tmp-dir\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.738139 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.738117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-home\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.738187 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.738169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-model-cache\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.738249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.738237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-tmp-dir\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.740203 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.740181    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-dshm\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.740380 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.740362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7537e768-3b42-436a-bc1b-a1d067fd30b7-tls-certs\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.745772 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.745751    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjzsh\" (UniqueName: \"kubernetes.io/projected/7537e768-3b42-436a-bc1b-a1d067fd30b7-kube-api-access-mjzsh\") pod \"e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") " pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:16.843433 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:16.843338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:17.118716 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:17.118681    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"]
Sep 09 20:32:17.120919 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:32:17.120894    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7537e768_3b42_436a_bc1b_a1d067fd30b7.slice/crio-0eece41c85a9bb7a10a050208c088e5ed48725de314b6017a6bf8bd373caab35 WatchSource:0}: Error finding container 0eece41c85a9bb7a10a050208c088e5ed48725de314b6017a6bf8bd373caab35: Status 404 returned error can't find the container with id 0eece41c85a9bb7a10a050208c088e5ed48725de314b6017a6bf8bd373caab35
Sep 09 20:32:17.886657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:17.886623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc" event={"ID":"7537e768-3b42-436a-bc1b-a1d067fd30b7","Type":"ContainerStarted","Data":"5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261"}
Sep 09 20:32:17.886657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:17.886664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc" event={"ID":"7537e768-3b42-436a-bc1b-a1d067fd30b7","Type":"ContainerStarted","Data":"0eece41c85a9bb7a10a050208c088e5ed48725de314b6017a6bf8bd373caab35"}
Sep 09 20:32:17.905068 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:17.905014    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc" podStartSLOduration=1.904996664 podStartE2EDuration="1.904996664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:32:17.902744254 +0000 UTC m=+959.395955238" watchObservedRunningTime="2026-09-09 20:32:17.904996664 +0000 UTC m=+959.398207648"
Sep 09 20:32:18.828251 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:18.828215    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf" podUID="4ea25009-4a2e-4c3b-ad08-eef3d899e272" containerName="main" containerID="cri-o://16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea" gracePeriod=45
Sep 09 20:32:19.096214 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.096192    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:32:19.157708 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.157678    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzwlg\" (UniqueName: \"kubernetes.io/projected/4ea25009-4a2e-4c3b-ad08-eef3d899e272-kube-api-access-dzwlg\") pod \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") "
Sep 09 20:32:19.157888 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.157716    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tmp-dir\") pod \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") "
Sep 09 20:32:19.157888 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.157742    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-dshm\") pod \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") "
Sep 09 20:32:19.157888 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.157780    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-home\") pod \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") "
Sep 09 20:32:19.157888 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.157819    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-model-cache\") pod \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") "
Sep 09 20:32:19.157888 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.157864    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tls-certs\") pod \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\" (UID: \"4ea25009-4a2e-4c3b-ad08-eef3d899e272\") "
Sep 09 20:32:19.158182 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.158069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-home" (OuterVolumeSpecName: "home") pod "4ea25009-4a2e-4c3b-ad08-eef3d899e272" (UID: "4ea25009-4a2e-4c3b-ad08-eef3d899e272"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:19.158182 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.158107    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ea25009-4a2e-4c3b-ad08-eef3d899e272" (UID: "4ea25009-4a2e-4c3b-ad08-eef3d899e272"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:19.158182 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.158160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ea25009-4a2e-4c3b-ad08-eef3d899e272" (UID: "4ea25009-4a2e-4c3b-ad08-eef3d899e272"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:19.158431 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.158254    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tmp-dir\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:32:19.158431 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.158274    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-home\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:32:19.158431 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.158287    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-model-cache\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:32:19.159964 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.159942    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-dshm" (OuterVolumeSpecName: "dshm") pod "4ea25009-4a2e-4c3b-ad08-eef3d899e272" (UID: "4ea25009-4a2e-4c3b-ad08-eef3d899e272"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:32:19.160082 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.160003    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea25009-4a2e-4c3b-ad08-eef3d899e272-kube-api-access-dzwlg" (OuterVolumeSpecName: "kube-api-access-dzwlg") pod "4ea25009-4a2e-4c3b-ad08-eef3d899e272" (UID: "4ea25009-4a2e-4c3b-ad08-eef3d899e272"). InnerVolumeSpecName "kube-api-access-dzwlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:32:19.160082 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.160038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ea25009-4a2e-4c3b-ad08-eef3d899e272" (UID: "4ea25009-4a2e-4c3b-ad08-eef3d899e272"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:32:19.259539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.259504    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwlg\" (UniqueName: \"kubernetes.io/projected/4ea25009-4a2e-4c3b-ad08-eef3d899e272-kube-api-access-dzwlg\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:32:19.259539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.259531    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ea25009-4a2e-4c3b-ad08-eef3d899e272-dshm\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:32:19.259539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.259541    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ea25009-4a2e-4c3b-ad08-eef3d899e272-tls-certs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:32:19.898988 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.898950    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ea25009-4a2e-4c3b-ad08-eef3d899e272" containerID="16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea" exitCode=0
Sep 09 20:32:19.899180 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.899010    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"
Sep 09 20:32:19.899180 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.899033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf" event={"ID":"4ea25009-4a2e-4c3b-ad08-eef3d899e272","Type":"ContainerDied","Data":"16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea"}
Sep 09 20:32:19.899180 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.899078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf" event={"ID":"4ea25009-4a2e-4c3b-ad08-eef3d899e272","Type":"ContainerDied","Data":"5cea6cc3a6fdec6e8b8a8d9909f790e3eed6b618a503321e6443db5059d8c322"}
Sep 09 20:32:19.899180 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.899098    2581 scope.go:117] "RemoveContainer" containerID="16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea"
Sep 09 20:32:19.908464 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.908434    2581 scope.go:117] "RemoveContainer" containerID="16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea"
Sep 09 20:32:19.908729 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:32:19.908710    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea\": container with ID starting with 16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea not found: ID does not exist" containerID="16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea"
Sep 09 20:32:19.908789 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.908737    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea"} err="failed to get container status \"16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea\": rpc error: code = NotFound desc = could not find container \"16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea\": container with ID starting with 16c5ca6c7121afa12568187ad7ae838beca3f97c9a677c604ba5d833a852d3ea not found: ID does not exist"
Sep 09 20:32:19.921021 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.920996    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"]
Sep 09 20:32:19.924600 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:19.924580    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-812760/autores-model-812760-kserve-744c7d6ff9-hs9xf"]
Sep 09 20:32:21.002862 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:21.002826    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea25009-4a2e-4c3b-ad08-eef3d899e272" path="/var/lib/kubelet/pods/4ea25009-4a2e-4c3b-ad08-eef3d899e272/volumes"
Sep 09 20:32:26.844302 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:26.844264    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:26.844810 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:26.844376    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:26.857185 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:26.857153    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:26.951332 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:26.951298    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:32:59.235587 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:32:59.235555    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"]
Sep 09 20:33:14.257274 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.257214    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc" podUID="7537e768-3b42-436a-bc1b-a1d067fd30b7" containerName="main" containerID="cri-o://5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261" gracePeriod=45
Sep 09 20:33:14.503557 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.503529    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:33:14.585515 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.585431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-home\") pod \"7537e768-3b42-436a-bc1b-a1d067fd30b7\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") "
Sep 09 20:33:14.585515 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.585467    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7537e768-3b42-436a-bc1b-a1d067fd30b7-tls-certs\") pod \"7537e768-3b42-436a-bc1b-a1d067fd30b7\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") "
Sep 09 20:33:14.585749 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.585537    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjzsh\" (UniqueName: \"kubernetes.io/projected/7537e768-3b42-436a-bc1b-a1d067fd30b7-kube-api-access-mjzsh\") pod \"7537e768-3b42-436a-bc1b-a1d067fd30b7\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") "
Sep 09 20:33:14.585749 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.585558    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-model-cache\") pod \"7537e768-3b42-436a-bc1b-a1d067fd30b7\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") "
Sep 09 20:33:14.585749 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.585598    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-tmp-dir\") pod \"7537e768-3b42-436a-bc1b-a1d067fd30b7\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") "
Sep 09 20:33:14.585749 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.585661    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-dshm\") pod \"7537e768-3b42-436a-bc1b-a1d067fd30b7\" (UID: \"7537e768-3b42-436a-bc1b-a1d067fd30b7\") "
Sep 09 20:33:14.585945 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.585750    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-home" (OuterVolumeSpecName: "home") pod "7537e768-3b42-436a-bc1b-a1d067fd30b7" (UID: "7537e768-3b42-436a-bc1b-a1d067fd30b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:14.585945 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.585842    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-model-cache" (OuterVolumeSpecName: "model-cache") pod "7537e768-3b42-436a-bc1b-a1d067fd30b7" (UID: "7537e768-3b42-436a-bc1b-a1d067fd30b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:14.585945 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.585909    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7537e768-3b42-436a-bc1b-a1d067fd30b7" (UID: "7537e768-3b42-436a-bc1b-a1d067fd30b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:14.586102 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.585952    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-model-cache\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.586102 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.585967    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-home\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.587894 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.587866    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-dshm" (OuterVolumeSpecName: "dshm") pod "7537e768-3b42-436a-bc1b-a1d067fd30b7" (UID: "7537e768-3b42-436a-bc1b-a1d067fd30b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:33:14.588000 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.587980    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7537e768-3b42-436a-bc1b-a1d067fd30b7-kube-api-access-mjzsh" (OuterVolumeSpecName: "kube-api-access-mjzsh") pod "7537e768-3b42-436a-bc1b-a1d067fd30b7" (UID: "7537e768-3b42-436a-bc1b-a1d067fd30b7"). InnerVolumeSpecName "kube-api-access-mjzsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:33:14.588051 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.587993    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7537e768-3b42-436a-bc1b-a1d067fd30b7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7537e768-3b42-436a-bc1b-a1d067fd30b7" (UID: "7537e768-3b42-436a-bc1b-a1d067fd30b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:33:14.687225 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.687189    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-dshm\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.687225 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.687221    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7537e768-3b42-436a-bc1b-a1d067fd30b7-tls-certs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.687225 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.687232    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjzsh\" (UniqueName: \"kubernetes.io/projected/7537e768-3b42-436a-bc1b-a1d067fd30b7-kube-api-access-mjzsh\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.687746 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:14.687241    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7537e768-3b42-436a-bc1b-a1d067fd30b7-tmp-dir\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:33:15.216231 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:15.216198    2581 generic.go:356] "Generic (PLEG): container finished" podID="7537e768-3b42-436a-bc1b-a1d067fd30b7" containerID="5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261" exitCode=0
Sep 09 20:33:15.216429 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:15.216243    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"
Sep 09 20:33:15.216429 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:15.216275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc" event={"ID":"7537e768-3b42-436a-bc1b-a1d067fd30b7","Type":"ContainerDied","Data":"5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261"}
Sep 09 20:33:15.216429 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:15.216317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc" event={"ID":"7537e768-3b42-436a-bc1b-a1d067fd30b7","Type":"ContainerDied","Data":"0eece41c85a9bb7a10a050208c088e5ed48725de314b6017a6bf8bd373caab35"}
Sep 09 20:33:15.216429 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:15.216335    2581 scope.go:117] "RemoveContainer" containerID="5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261"
Sep 09 20:33:15.225546 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:15.225518    2581 scope.go:117] "RemoveContainer" containerID="5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261"
Sep 09 20:33:15.225805 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:33:15.225788    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261\": container with ID starting with 5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261 not found: ID does not exist" containerID="5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261"
Sep 09 20:33:15.225856 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:15.225813    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261"} err="failed to get container status \"5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261\": rpc error: code = NotFound desc = could not find container \"5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261\": container with ID starting with 5ad81e1b41de2da074fd83a236dbb5dc1f08bdf43ebe8f8bbca1da6d81cca261 not found: ID does not exist"
Sep 09 20:33:15.233807 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:15.233784    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"]
Sep 09 20:33:15.236110 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:15.236090    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-836c5d71-kserve-75b67f78f4-f6wsc"]
Sep 09 20:33:17.008551 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:17.008517    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7537e768-3b42-436a-bc1b-a1d067fd30b7" path="/var/lib/kubelet/pods/7537e768-3b42-436a-bc1b-a1d067fd30b7/volumes"
Sep 09 20:33:30.247033 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.246993    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"]
Sep 09 20:33:30.247517 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.247406    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7537e768-3b42-436a-bc1b-a1d067fd30b7" containerName="main"
Sep 09 20:33:30.247517 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.247419    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7537e768-3b42-436a-bc1b-a1d067fd30b7" containerName="main"
Sep 09 20:33:30.247517 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.247429    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea25009-4a2e-4c3b-ad08-eef3d899e272" containerName="main"
Sep 09 20:33:30.247517 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.247434    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea25009-4a2e-4c3b-ad08-eef3d899e272" containerName="main"
Sep 09 20:33:30.247517 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.247491    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7537e768-3b42-436a-bc1b-a1d067fd30b7" containerName="main"
Sep 09 20:33:30.247517 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.247499    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea25009-4a2e-4c3b-ad08-eef3d899e272" containerName="main"
Sep 09 20:33:30.251964 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.251946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.255504 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.255481    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4defd2\"/\"test-model-4defd2-kserve-self-signed-certs\""
Sep 09 20:33:30.255653 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.255516    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4defd2\"/\"openshift-service-ca.crt\""
Sep 09 20:33:30.255653 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.255567    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4defd2\"/\"default-dockercfg-d8j8f\""
Sep 09 20:33:30.255653 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.255581    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4defd2\"/\"kube-root-ca.crt\""
Sep 09 20:33:30.261027 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.261000    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"]
Sep 09 20:33:30.319077 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.319036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5hhk\" (UniqueName: \"kubernetes.io/projected/e50b7107-d192-4cdc-8358-0b7dd4096bd1-kube-api-access-w5hhk\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.319248 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.319136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tls-certs\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.319248 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.319180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-model-cache\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.319248 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.319241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-home\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.319386 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.319266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tmp-dir\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.319386 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.319332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-dshm\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.420104 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.420074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-dshm\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.420282 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.420115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5hhk\" (UniqueName: \"kubernetes.io/projected/e50b7107-d192-4cdc-8358-0b7dd4096bd1-kube-api-access-w5hhk\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.420282 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.420145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tls-certs\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.420282 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.420180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-model-cache\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.420282 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.420237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-home\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.420282 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.420276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tmp-dir\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.420627 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.420588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-home\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.420627 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.420617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-model-cache\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.420700 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.420682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tmp-dir\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.422523 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.422502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-dshm\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.422831 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.422811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tls-certs\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.427868 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.427846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5hhk\" (UniqueName: \"kubernetes.io/projected/e50b7107-d192-4cdc-8358-0b7dd4096bd1-kube-api-access-w5hhk\") pod \"test-model-4defd2-kserve-569998667-69xwp\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") " pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.562500 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.562417    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:30.840126 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:30.840097    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"]
Sep 09 20:33:30.841631 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:33:30.841601    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode50b7107_d192_4cdc_8358_0b7dd4096bd1.slice/crio-818e6be25f0181c8691fae3327ab83c39ef1bfa3f32cb6f44e6d152c2d293d47 WatchSource:0}: Error finding container 818e6be25f0181c8691fae3327ab83c39ef1bfa3f32cb6f44e6d152c2d293d47: Status 404 returned error can't find the container with id 818e6be25f0181c8691fae3327ab83c39ef1bfa3f32cb6f44e6d152c2d293d47
Sep 09 20:33:31.306024 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:31.305980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp" event={"ID":"e50b7107-d192-4cdc-8358-0b7dd4096bd1","Type":"ContainerStarted","Data":"15194b95cad30322715d308560432b6e36da28044a0cbef8523738168f5ebde0"}
Sep 09 20:33:31.306024 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:31.306021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp" event={"ID":"e50b7107-d192-4cdc-8358-0b7dd4096bd1","Type":"ContainerStarted","Data":"818e6be25f0181c8691fae3327ab83c39ef1bfa3f32cb6f44e6d152c2d293d47"}
Sep 09 20:33:31.324448 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:31.324392    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp" podStartSLOduration=1.32437531 podStartE2EDuration="1.32437531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:31.32217103 +0000 UTC m=+1032.815382015" watchObservedRunningTime="2026-09-09 20:33:31.32437531 +0000 UTC m=+1032.817586293"
Sep 09 20:33:40.563426 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:40.563387    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:40.563844 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:40.563442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:40.576690 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:40.576659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:33:41.373497 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:33:41.373471    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:34:23.289304 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:23.289219    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"]
Sep 09 20:34:23.292978 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:23.292960    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"
Sep 09 20:34:23.296585 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:23.296562    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:34:23.296919 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:23.296897    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:34:23.306638 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:23.306613    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"]
Sep 09 20:34:23.321775 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:23.321752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c83915c-3599-4152-87f4-179027f0aed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-08bc01-d82hr\" (UID: \"9c83915c-3599-4152-87f4-179027f0aed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"
Sep 09 20:34:23.422923 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:23.422887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c83915c-3599-4152-87f4-179027f0aed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-08bc01-d82hr\" (UID: \"9c83915c-3599-4152-87f4-179027f0aed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"
Sep 09 20:34:23.423570 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:23.423550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c83915c-3599-4152-87f4-179027f0aed5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-08bc01-d82hr\" (UID: \"9c83915c-3599-4152-87f4-179027f0aed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"
Sep 09 20:34:23.603976 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:23.603886    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"
Sep 09 20:34:23.914370 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:23.914327    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"]
Sep 09 20:34:23.915862 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:34:23.915831    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c83915c_3599_4152_87f4_179027f0aed5.slice/crio-4efe8f9b11a91662481a0ec033836e133ac35836e82a53cf1d0606abf733cd12 WatchSource:0}: Error finding container 4efe8f9b11a91662481a0ec033836e133ac35836e82a53cf1d0606abf733cd12: Status 404 returned error can't find the container with id 4efe8f9b11a91662481a0ec033836e133ac35836e82a53cf1d0606abf733cd12
Sep 09 20:34:24.612808 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.612750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr" event={"ID":"9c83915c-3599-4152-87f4-179027f0aed5","Type":"ContainerStarted","Data":"4addc4ec335d7e63f3aee94b77794ca0f81be92d5a429ae782b3b794a5489918"}
Sep 09 20:34:24.612808 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.612803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr" event={"ID":"9c83915c-3599-4152-87f4-179027f0aed5","Type":"ContainerStarted","Data":"4efe8f9b11a91662481a0ec033836e133ac35836e82a53cf1d0606abf733cd12"}
Sep 09 20:34:24.632081 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.632032    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr" podStartSLOduration=1.632017292 podStartE2EDuration="1.632017292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:24.630546825 +0000 UTC m=+1086.123757806" watchObservedRunningTime="2026-09-09 20:34:24.632017292 +0000 UTC m=+1086.125228276"
Sep 09 20:34:24.696888 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.696859    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"]
Sep 09 20:34:24.700234 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.700217    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:34:24.703412 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.703391    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-36f068b3\""
Sep 09 20:34:24.703662 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.703643    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-36f068b3-dockercfg-5bhx7\""
Sep 09 20:34:24.713608 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.713585    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"]
Sep 09 20:34:24.730661 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.730636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp\" (UID: \"ab24bbf6-3f46-45bd-9d9c-d08d024e5079\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:34:24.730793 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.730680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n7lp\" (UniqueName: \"kubernetes.io/projected/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-kube-api-access-6n7lp\") pod \"payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp\" (UID: \"ab24bbf6-3f46-45bd-9d9c-d08d024e5079\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:34:24.831214 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.831182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp\" (UID: \"ab24bbf6-3f46-45bd-9d9c-d08d024e5079\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:34:24.831435 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.831228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n7lp\" (UniqueName: \"kubernetes.io/projected/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-kube-api-access-6n7lp\") pod \"payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp\" (UID: \"ab24bbf6-3f46-45bd-9d9c-d08d024e5079\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:34:24.831940 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.831915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp\" (UID: \"ab24bbf6-3f46-45bd-9d9c-d08d024e5079\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:34:24.841036 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:24.841003    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n7lp\" (UniqueName: \"kubernetes.io/projected/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-kube-api-access-6n7lp\") pod \"payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp\" (UID: \"ab24bbf6-3f46-45bd-9d9c-d08d024e5079\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:34:25.010907 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:25.010867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:34:25.295722 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:25.295697    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"]
Sep 09 20:34:25.297581 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:34:25.297551    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab24bbf6_3f46_45bd_9d9c_d08d024e5079.slice/crio-1fcfc6c8c2100fa0f72ac1ebc34db4ba69751de7d4579c919d31fd29b8ff6f58 WatchSource:0}: Error finding container 1fcfc6c8c2100fa0f72ac1ebc34db4ba69751de7d4579c919d31fd29b8ff6f58: Status 404 returned error can't find the container with id 1fcfc6c8c2100fa0f72ac1ebc34db4ba69751de7d4579c919d31fd29b8ff6f58
Sep 09 20:34:25.620198 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:25.620101    2581 generic.go:356] "Generic (PLEG): container finished" podID="9c83915c-3599-4152-87f4-179027f0aed5" containerID="4addc4ec335d7e63f3aee94b77794ca0f81be92d5a429ae782b3b794a5489918" exitCode=0
Sep 09 20:34:25.620198 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:25.620168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr" event={"ID":"9c83915c-3599-4152-87f4-179027f0aed5","Type":"ContainerDied","Data":"4addc4ec335d7e63f3aee94b77794ca0f81be92d5a429ae782b3b794a5489918"}
Sep 09 20:34:25.621970 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:25.621943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp" event={"ID":"ab24bbf6-3f46-45bd-9d9c-d08d024e5079","Type":"ContainerStarted","Data":"ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7"}
Sep 09 20:34:25.622107 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:25.621985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp" event={"ID":"ab24bbf6-3f46-45bd-9d9c-d08d024e5079","Type":"ContainerStarted","Data":"1fcfc6c8c2100fa0f72ac1ebc34db4ba69751de7d4579c919d31fd29b8ff6f58"}
Sep 09 20:34:25.622107 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:25.622005    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:34:25.674286 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:25.674236    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp" podStartSLOduration=1.6742226009999999 podStartE2EDuration="1.674222601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:34:25.669290458 +0000 UTC m=+1087.162501441" watchObservedRunningTime="2026-09-09 20:34:25.674222601 +0000 UTC m=+1087.167433584"
Sep 09 20:34:26.773242 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:26.773218    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"
Sep 09 20:34:26.844411 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:26.844380    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c83915c-3599-4152-87f4-179027f0aed5-maas-api-service-ca\") pod \"9c83915c-3599-4152-87f4-179027f0aed5\" (UID: \"9c83915c-3599-4152-87f4-179027f0aed5\") "
Sep 09 20:34:26.844686 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:26.844663    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c83915c-3599-4152-87f4-179027f0aed5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c83915c-3599-4152-87f4-179027f0aed5" (UID: "9c83915c-3599-4152-87f4-179027f0aed5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:26.945337 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:26.945302    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c83915c-3599-4152-87f4-179027f0aed5-maas-api-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:34:27.634945 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:27.634912    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"
Sep 09 20:34:27.635139 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:27.634955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr" event={"ID":"9c83915c-3599-4152-87f4-179027f0aed5","Type":"ContainerDied","Data":"4efe8f9b11a91662481a0ec033836e133ac35836e82a53cf1d0606abf733cd12"}
Sep 09 20:34:27.635139 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:27.635046    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4efe8f9b11a91662481a0ec033836e133ac35836e82a53cf1d0606abf733cd12"
Sep 09 20:34:34.486109 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:34.486071    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"]
Sep 09 20:34:34.489380 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:34.489326    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-08bc01-d82hr"]
Sep 09 20:34:35.002768 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:35.002723    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c83915c-3599-4152-87f4-179027f0aed5" path="/var/lib/kubelet/pods/9c83915c-3599-4152-87f4-179027f0aed5/volumes"
Sep 09 20:34:36.629711 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:36.629682    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:34:49.252588 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.252549    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"]
Sep 09 20:34:49.252999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.252956    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c83915c-3599-4152-87f4-179027f0aed5" containerName="revoke-keys"
Sep 09 20:34:49.252999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.252968    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c83915c-3599-4152-87f4-179027f0aed5" containerName="revoke-keys"
Sep 09 20:34:49.253072 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.253032    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c83915c-3599-4152-87f4-179027f0aed5" containerName="revoke-keys"
Sep 09 20:34:49.256185 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.256167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"
Sep 09 20:34:49.261440 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.261400    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:34:49.262374 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.262321    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:34:49.266214 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.266188    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"]
Sep 09 20:34:49.347407 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.347369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b75e187-2af4-4d21-bf81-25e196757820-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn\" (UID: \"5b75e187-2af4-4d21-bf81-25e196757820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"
Sep 09 20:34:49.448811 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.448768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b75e187-2af4-4d21-bf81-25e196757820-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn\" (UID: \"5b75e187-2af4-4d21-bf81-25e196757820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"
Sep 09 20:34:49.449478 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.449456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b75e187-2af4-4d21-bf81-25e196757820-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn\" (UID: \"5b75e187-2af4-4d21-bf81-25e196757820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"
Sep 09 20:34:49.569293 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.569205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"
Sep 09 20:34:49.859237 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:49.859205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"]
Sep 09 20:34:49.860317 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:34:49.860284    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b75e187_2af4_4d21_bf81_25e196757820.slice/crio-fdbb9907ca3482cc72dd16dcbafc121d7fc63141c26da7425f1f5ec0dde49aac WatchSource:0}: Error finding container fdbb9907ca3482cc72dd16dcbafc121d7fc63141c26da7425f1f5ec0dde49aac: Status 404 returned error can't find the container with id fdbb9907ca3482cc72dd16dcbafc121d7fc63141c26da7425f1f5ec0dde49aac
Sep 09 20:34:50.780736 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:50.780695    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b75e187-2af4-4d21-bf81-25e196757820" containerID="1116704d65418f90e030d185e1a60e07e2f903d079da5b610499a79977cc8028" exitCode=0
Sep 09 20:34:50.781190 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:50.780785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn" event={"ID":"5b75e187-2af4-4d21-bf81-25e196757820","Type":"ContainerDied","Data":"1116704d65418f90e030d185e1a60e07e2f903d079da5b610499a79977cc8028"}
Sep 09 20:34:50.781190 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:50.780832    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn" event={"ID":"5b75e187-2af4-4d21-bf81-25e196757820","Type":"ContainerStarted","Data":"fdbb9907ca3482cc72dd16dcbafc121d7fc63141c26da7425f1f5ec0dde49aac"}
Sep 09 20:34:51.930455 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:51.930430    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"
Sep 09 20:34:52.073594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:52.073490    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b75e187-2af4-4d21-bf81-25e196757820-maas-api-service-ca\") pod \"5b75e187-2af4-4d21-bf81-25e196757820\" (UID: \"5b75e187-2af4-4d21-bf81-25e196757820\") "
Sep 09 20:34:52.073883 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:52.073857    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b75e187-2af4-4d21-bf81-25e196757820-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b75e187-2af4-4d21-bf81-25e196757820" (UID: "5b75e187-2af4-4d21-bf81-25e196757820"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:34:52.174964 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:52.174924    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b75e187-2af4-4d21-bf81-25e196757820-maas-api-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:34:52.794556 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:52.794524    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"
Sep 09 20:34:52.794739 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:52.794527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn" event={"ID":"5b75e187-2af4-4d21-bf81-25e196757820","Type":"ContainerDied","Data":"fdbb9907ca3482cc72dd16dcbafc121d7fc63141c26da7425f1f5ec0dde49aac"}
Sep 09 20:34:52.794739 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:34:52.794653    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdbb9907ca3482cc72dd16dcbafc121d7fc63141c26da7425f1f5ec0dde49aac"
Sep 09 20:35:02.303308 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.303275    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"]
Sep 09 20:35:02.303804 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.303565    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp" podUID="ab24bbf6-3f46-45bd-9d9c-d08d024e5079" containerName="payload-pre-processing" containerID="cri-o://ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7" gracePeriod=30
Sep 09 20:35:02.468300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.468245    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"]
Sep 09 20:35:02.470637 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.470616    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-36f068b3-fhhgn"]
Sep 09 20:35:02.570641 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.570616    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:35:02.667907 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.667875    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-plugins-config-volume\") pod \"ab24bbf6-3f46-45bd-9d9c-d08d024e5079\" (UID: \"ab24bbf6-3f46-45bd-9d9c-d08d024e5079\") "
Sep 09 20:35:02.668103 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.667956    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n7lp\" (UniqueName: \"kubernetes.io/projected/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-kube-api-access-6n7lp\") pod \"ab24bbf6-3f46-45bd-9d9c-d08d024e5079\" (UID: \"ab24bbf6-3f46-45bd-9d9c-d08d024e5079\") "
Sep 09 20:35:02.668261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.668236    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab24bbf6-3f46-45bd-9d9c-d08d024e5079" (UID: "ab24bbf6-3f46-45bd-9d9c-d08d024e5079"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:02.670193 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.670171    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-kube-api-access-6n7lp" (OuterVolumeSpecName: "kube-api-access-6n7lp") pod "ab24bbf6-3f46-45bd-9d9c-d08d024e5079" (UID: "ab24bbf6-3f46-45bd-9d9c-d08d024e5079"). InnerVolumeSpecName "kube-api-access-6n7lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:02.769530 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.769499    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:02.769530 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.769526    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n7lp\" (UniqueName: \"kubernetes.io/projected/ab24bbf6-3f46-45bd-9d9c-d08d024e5079-kube-api-access-6n7lp\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:02.851384 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.851270    2581 generic.go:356] "Generic (PLEG): container finished" podID="ab24bbf6-3f46-45bd-9d9c-d08d024e5079" containerID="ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7" exitCode=0
Sep 09 20:35:02.851384 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.851317    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"
Sep 09 20:35:02.851384 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.851368    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp" event={"ID":"ab24bbf6-3f46-45bd-9d9c-d08d024e5079","Type":"ContainerDied","Data":"ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7"}
Sep 09 20:35:02.851643 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.851419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp" event={"ID":"ab24bbf6-3f46-45bd-9d9c-d08d024e5079","Type":"ContainerDied","Data":"1fcfc6c8c2100fa0f72ac1ebc34db4ba69751de7d4579c919d31fd29b8ff6f58"}
Sep 09 20:35:02.851643 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.851441    2581 scope.go:117] "RemoveContainer" containerID="ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7"
Sep 09 20:35:02.861460 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.861438    2581 scope.go:117] "RemoveContainer" containerID="ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7"
Sep 09 20:35:02.861749 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:35:02.861729    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7\": container with ID starting with ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7 not found: ID does not exist" containerID="ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7"
Sep 09 20:35:02.861803 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.861758    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7"} err="failed to get container status \"ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7\": rpc error: code = NotFound desc = could not find container \"ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7\": container with ID starting with ae23e2102b530897cf42c4d1e2402bf04e87d11d25bc7a04571eacd28b5ac9d7 not found: ID does not exist"
Sep 09 20:35:02.891617 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.891589    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"]
Sep 09 20:35:02.895914 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:02.895893    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-36f068b3-74b948cd58-5psjp"]
Sep 09 20:35:03.002142 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:03.002112    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b75e187-2af4-4d21-bf81-25e196757820" path="/var/lib/kubelet/pods/5b75e187-2af4-4d21-bf81-25e196757820/volumes"
Sep 09 20:35:03.002496 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:03.002482    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab24bbf6-3f46-45bd-9d9c-d08d024e5079" path="/var/lib/kubelet/pods/ab24bbf6-3f46-45bd-9d9c-d08d024e5079/volumes"
Sep 09 20:35:16.854405 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:16.854370    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"]
Sep 09 20:35:17.840137 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:17.840103    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"]
Sep 09 20:35:31.008167 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.008131    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"]
Sep 09 20:35:31.008698 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.008431    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h" podUID="c683389c-4524-41e6-9501-a649e694a6d6" containerName="payload-processing" containerID="cri-o://303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3" gracePeriod=30
Sep 09 20:35:31.013884 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.013855    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"]
Sep 09 20:35:31.014255 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.014225    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx" podUID="cc9b9f9f-aa25-4398-ae65-abf3a644f103" containerName="payload-pre-processing" containerID="cri-o://263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f" gracePeriod=30
Sep 09 20:35:31.269518 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.269443    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:35:31.300225 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.300196    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:35:31.331889 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.331862    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kjn2\" (UniqueName: \"kubernetes.io/projected/c683389c-4524-41e6-9501-a649e694a6d6-kube-api-access-8kjn2\") pod \"c683389c-4524-41e6-9501-a649e694a6d6\" (UID: \"c683389c-4524-41e6-9501-a649e694a6d6\") "
Sep 09 20:35:31.332087 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.331976    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c683389c-4524-41e6-9501-a649e694a6d6-plugins-config-volume\") pod \"c683389c-4524-41e6-9501-a649e694a6d6\" (UID: \"c683389c-4524-41e6-9501-a649e694a6d6\") "
Sep 09 20:35:31.332575 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.332418    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c683389c-4524-41e6-9501-a649e694a6d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c683389c-4524-41e6-9501-a649e694a6d6" (UID: "c683389c-4524-41e6-9501-a649e694a6d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:31.334327 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.334301    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c683389c-4524-41e6-9501-a649e694a6d6-kube-api-access-8kjn2" (OuterVolumeSpecName: "kube-api-access-8kjn2") pod "c683389c-4524-41e6-9501-a649e694a6d6" (UID: "c683389c-4524-41e6-9501-a649e694a6d6"). InnerVolumeSpecName "kube-api-access-8kjn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:31.433019 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.432989    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk6tm\" (UniqueName: \"kubernetes.io/projected/cc9b9f9f-aa25-4398-ae65-abf3a644f103-kube-api-access-sk6tm\") pod \"cc9b9f9f-aa25-4398-ae65-abf3a644f103\" (UID: \"cc9b9f9f-aa25-4398-ae65-abf3a644f103\") "
Sep 09 20:35:31.433227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.433101    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9b9f9f-aa25-4398-ae65-abf3a644f103-plugins-config-volume\") pod \"cc9b9f9f-aa25-4398-ae65-abf3a644f103\" (UID: \"cc9b9f9f-aa25-4398-ae65-abf3a644f103\") "
Sep 09 20:35:31.433411 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.433391    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c683389c-4524-41e6-9501-a649e694a6d6-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:31.433489 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.433414    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kjn2\" (UniqueName: \"kubernetes.io/projected/c683389c-4524-41e6-9501-a649e694a6d6-kube-api-access-8kjn2\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:31.433489 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.433430    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9b9f9f-aa25-4398-ae65-abf3a644f103-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc9b9f9f-aa25-4398-ae65-abf3a644f103" (UID: "cc9b9f9f-aa25-4398-ae65-abf3a644f103"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:31.435232 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.435209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9b9f9f-aa25-4398-ae65-abf3a644f103-kube-api-access-sk6tm" (OuterVolumeSpecName: "kube-api-access-sk6tm") pod "cc9b9f9f-aa25-4398-ae65-abf3a644f103" (UID: "cc9b9f9f-aa25-4398-ae65-abf3a644f103"). InnerVolumeSpecName "kube-api-access-sk6tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:31.534851 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.534764    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk6tm\" (UniqueName: \"kubernetes.io/projected/cc9b9f9f-aa25-4398-ae65-abf3a644f103-kube-api-access-sk6tm\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:31.534851 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.534794    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc9b9f9f-aa25-4398-ae65-abf3a644f103-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:31.874457 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:31.874359    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb" podUID="496a7626-e8a3-4ade-9138-ae2123dfd840" containerName="main" containerID="cri-o://d6818a4104054e7603b13a2aef050b2ac3a8b3a0737616c10be698cdac90594d" gracePeriod=45
Sep 09 20:35:32.031090 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.030483    2581 generic.go:356] "Generic (PLEG): container finished" podID="496a7626-e8a3-4ade-9138-ae2123dfd840" containerID="d6818a4104054e7603b13a2aef050b2ac3a8b3a0737616c10be698cdac90594d" exitCode=0
Sep 09 20:35:32.031090 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.030559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb" event={"ID":"496a7626-e8a3-4ade-9138-ae2123dfd840","Type":"ContainerDied","Data":"d6818a4104054e7603b13a2aef050b2ac3a8b3a0737616c10be698cdac90594d"}
Sep 09 20:35:32.032234 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.032210    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc9b9f9f-aa25-4398-ae65-abf3a644f103" containerID="263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f" exitCode=0
Sep 09 20:35:32.032502 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.032272    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"
Sep 09 20:35:32.032502 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.032296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx" event={"ID":"cc9b9f9f-aa25-4398-ae65-abf3a644f103","Type":"ContainerDied","Data":"263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f"}
Sep 09 20:35:32.032502 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.032397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx" event={"ID":"cc9b9f9f-aa25-4398-ae65-abf3a644f103","Type":"ContainerDied","Data":"5689fd63c9beb6365a86f04d654d33c10d5b767d95c6777cc4d7e9217ae55f1c"}
Sep 09 20:35:32.032699 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.032508    2581 scope.go:117] "RemoveContainer" containerID="263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f"
Sep 09 20:35:32.034032 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.034009    2581 generic.go:356] "Generic (PLEG): container finished" podID="c683389c-4524-41e6-9501-a649e694a6d6" containerID="303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3" exitCode=0
Sep 09 20:35:32.034107 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.034066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h" event={"ID":"c683389c-4524-41e6-9501-a649e694a6d6","Type":"ContainerDied","Data":"303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3"}
Sep 09 20:35:32.034166 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.034104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h" event={"ID":"c683389c-4524-41e6-9501-a649e694a6d6","Type":"ContainerDied","Data":"8ddd88e38f2942a2def325588851f6b156df3d72d8569354e4d42913aa8d0171"}
Sep 09 20:35:32.034227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.034181    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"
Sep 09 20:35:32.047006 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.046461    2581 scope.go:117] "RemoveContainer" containerID="263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f"
Sep 09 20:35:32.047006 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:35:32.046931    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f\": container with ID starting with 263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f not found: ID does not exist" containerID="263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f"
Sep 09 20:35:32.047006 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.046964    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f"} err="failed to get container status \"263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f\": rpc error: code = NotFound desc = could not find container \"263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f\": container with ID starting with 263a2ac09828fa24e733e74649203552492e1219548b919cc40fc4c4e1a5fa2f not found: ID does not exist"
Sep 09 20:35:32.047006 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.046986    2581 scope.go:117] "RemoveContainer" containerID="303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3"
Sep 09 20:35:32.057814 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.057795    2581 scope.go:117] "RemoveContainer" containerID="303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3"
Sep 09 20:35:32.058177 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:35:32.058151    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3\": container with ID starting with 303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3 not found: ID does not exist" containerID="303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3"
Sep 09 20:35:32.058246 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.058188    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3"} err="failed to get container status \"303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3\": rpc error: code = NotFound desc = could not find container \"303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3\": container with ID starting with 303f1fe75291e7dcbab371ba73b3c28f60d14a4ad77297ae238e009899b7dfc3 not found: ID does not exist"
Sep 09 20:35:32.069305 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.069235    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"]
Sep 09 20:35:32.074984 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.074959    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-21117b-7dc6d694f4-92rtx"]
Sep 09 20:35:32.084786 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.084763    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"]
Sep 09 20:35:32.088259 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.088235    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-21117b-747669564f-vgc5h"]
Sep 09 20:35:32.137711 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.137653    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:35:32.240644 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.240605    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-model-cache\") pod \"496a7626-e8a3-4ade-9138-ae2123dfd840\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") "
Sep 09 20:35:32.240644 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.240641    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-tmp-dir\") pod \"496a7626-e8a3-4ade-9138-ae2123dfd840\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") "
Sep 09 20:35:32.240866 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.240671    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65hgp\" (UniqueName: \"kubernetes.io/projected/496a7626-e8a3-4ade-9138-ae2123dfd840-kube-api-access-65hgp\") pod \"496a7626-e8a3-4ade-9138-ae2123dfd840\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") "
Sep 09 20:35:32.240866 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.240721    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-dshm\") pod \"496a7626-e8a3-4ade-9138-ae2123dfd840\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") "
Sep 09 20:35:32.240866 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.240751    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496a7626-e8a3-4ade-9138-ae2123dfd840-tls-certs\") pod \"496a7626-e8a3-4ade-9138-ae2123dfd840\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") "
Sep 09 20:35:32.241055 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.240892    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-home\") pod \"496a7626-e8a3-4ade-9138-ae2123dfd840\" (UID: \"496a7626-e8a3-4ade-9138-ae2123dfd840\") "
Sep 09 20:35:32.241055 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.240966    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-model-cache" (OuterVolumeSpecName: "model-cache") pod "496a7626-e8a3-4ade-9138-ae2123dfd840" (UID: "496a7626-e8a3-4ade-9138-ae2123dfd840"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:32.241055 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.240984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "496a7626-e8a3-4ade-9138-ae2123dfd840" (UID: "496a7626-e8a3-4ade-9138-ae2123dfd840"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:32.241209 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.241115    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-home" (OuterVolumeSpecName: "home") pod "496a7626-e8a3-4ade-9138-ae2123dfd840" (UID: "496a7626-e8a3-4ade-9138-ae2123dfd840"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:32.241296 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.241277    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-home\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:32.241338 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.241303    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-model-cache\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:32.241408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.241369    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-tmp-dir\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:32.243179 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.243153    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-dshm" (OuterVolumeSpecName: "dshm") pod "496a7626-e8a3-4ade-9138-ae2123dfd840" (UID: "496a7626-e8a3-4ade-9138-ae2123dfd840"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:32.243324 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.243205    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496a7626-e8a3-4ade-9138-ae2123dfd840-kube-api-access-65hgp" (OuterVolumeSpecName: "kube-api-access-65hgp") pod "496a7626-e8a3-4ade-9138-ae2123dfd840" (UID: "496a7626-e8a3-4ade-9138-ae2123dfd840"). InnerVolumeSpecName "kube-api-access-65hgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:32.243324 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.243238    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496a7626-e8a3-4ade-9138-ae2123dfd840-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "496a7626-e8a3-4ade-9138-ae2123dfd840" (UID: "496a7626-e8a3-4ade-9138-ae2123dfd840"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:32.342071 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.342021    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65hgp\" (UniqueName: \"kubernetes.io/projected/496a7626-e8a3-4ade-9138-ae2123dfd840-kube-api-access-65hgp\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:32.342071 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.342068    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496a7626-e8a3-4ade-9138-ae2123dfd840-dshm\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:32.342071 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.342078    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496a7626-e8a3-4ade-9138-ae2123dfd840-tls-certs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:32.860153 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:32.860096    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp" podUID="e50b7107-d192-4cdc-8358-0b7dd4096bd1" containerName="main" containerID="cri-o://15194b95cad30322715d308560432b6e36da28044a0cbef8523738168f5ebde0" gracePeriod=45
Sep 09 20:35:33.002549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.002514    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c683389c-4524-41e6-9501-a649e694a6d6" path="/var/lib/kubelet/pods/c683389c-4524-41e6-9501-a649e694a6d6/volumes"
Sep 09 20:35:33.002882 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.002866    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9b9f9f-aa25-4398-ae65-abf3a644f103" path="/var/lib/kubelet/pods/cc9b9f9f-aa25-4398-ae65-abf3a644f103/volumes"
Sep 09 20:35:33.041738 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.041705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb" event={"ID":"496a7626-e8a3-4ade-9138-ae2123dfd840","Type":"ContainerDied","Data":"a19734e16de68bb23c0595f199ed726f61d23fc5afbb1bde7bc10530645102fd"}
Sep 09 20:35:33.041738 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.041734    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"
Sep 09 20:35:33.042241 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.041770    2581 scope.go:117] "RemoveContainer" containerID="d6818a4104054e7603b13a2aef050b2ac3a8b3a0737616c10be698cdac90594d"
Sep 09 20:35:33.044090 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.044049    2581 generic.go:356] "Generic (PLEG): container finished" podID="e50b7107-d192-4cdc-8358-0b7dd4096bd1" containerID="15194b95cad30322715d308560432b6e36da28044a0cbef8523738168f5ebde0" exitCode=0
Sep 09 20:35:33.044510 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.044184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp" event={"ID":"e50b7107-d192-4cdc-8358-0b7dd4096bd1","Type":"ContainerDied","Data":"15194b95cad30322715d308560432b6e36da28044a0cbef8523738168f5ebde0"}
Sep 09 20:35:33.063582 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.063555    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"]
Sep 09 20:35:33.067866 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.067844    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-21117b/test-model-21117b-kserve-67c54df97c-tk5sb"]
Sep 09 20:35:33.123843 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.123781    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:35:33.249445 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.249411    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tmp-dir\") pod \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") "
Sep 09 20:35:33.249646 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.249474    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5hhk\" (UniqueName: \"kubernetes.io/projected/e50b7107-d192-4cdc-8358-0b7dd4096bd1-kube-api-access-w5hhk\") pod \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") "
Sep 09 20:35:33.249646 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.249507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-dshm\") pod \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") "
Sep 09 20:35:33.249646 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.249611    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-home\") pod \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") "
Sep 09 20:35:33.249818 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.249657    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-model-cache\") pod \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") "
Sep 09 20:35:33.249818 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.249698    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tls-certs\") pod \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\" (UID: \"e50b7107-d192-4cdc-8358-0b7dd4096bd1\") "
Sep 09 20:35:33.249818 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.249733    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e50b7107-d192-4cdc-8358-0b7dd4096bd1" (UID: "e50b7107-d192-4cdc-8358-0b7dd4096bd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:33.250019 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.249872    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-home" (OuterVolumeSpecName: "home") pod "e50b7107-d192-4cdc-8358-0b7dd4096bd1" (UID: "e50b7107-d192-4cdc-8358-0b7dd4096bd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:33.250019 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.249888    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "e50b7107-d192-4cdc-8358-0b7dd4096bd1" (UID: "e50b7107-d192-4cdc-8358-0b7dd4096bd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:33.250089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.250071    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-model-cache\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.250089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.250087    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tmp-dir\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.250169 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.250096    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-home\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.251979 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.251947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e50b7107-d192-4cdc-8358-0b7dd4096bd1" (UID: "e50b7107-d192-4cdc-8358-0b7dd4096bd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:35:33.252193 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.252170    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-dshm" (OuterVolumeSpecName: "dshm") pod "e50b7107-d192-4cdc-8358-0b7dd4096bd1" (UID: "e50b7107-d192-4cdc-8358-0b7dd4096bd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:35:33.252193 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.252189    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50b7107-d192-4cdc-8358-0b7dd4096bd1-kube-api-access-w5hhk" (OuterVolumeSpecName: "kube-api-access-w5hhk") pod "e50b7107-d192-4cdc-8358-0b7dd4096bd1" (UID: "e50b7107-d192-4cdc-8358-0b7dd4096bd1"). InnerVolumeSpecName "kube-api-access-w5hhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:33.351614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.351576    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5hhk\" (UniqueName: \"kubernetes.io/projected/e50b7107-d192-4cdc-8358-0b7dd4096bd1-kube-api-access-w5hhk\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.351614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.351604    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50b7107-d192-4cdc-8358-0b7dd4096bd1-dshm\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:33.351614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:33.351615    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50b7107-d192-4cdc-8358-0b7dd4096bd1-tls-certs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:34.055144 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:34.055119    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"
Sep 09 20:35:34.055622 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:34.055115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp" event={"ID":"e50b7107-d192-4cdc-8358-0b7dd4096bd1","Type":"ContainerDied","Data":"818e6be25f0181c8691fae3327ab83c39ef1bfa3f32cb6f44e6d152c2d293d47"}
Sep 09 20:35:34.055622 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:34.055246    2581 scope.go:117] "RemoveContainer" containerID="15194b95cad30322715d308560432b6e36da28044a0cbef8523738168f5ebde0"
Sep 09 20:35:34.079478 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:34.079451    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"]
Sep 09 20:35:34.083542 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:34.083521    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4defd2/test-model-4defd2-kserve-569998667-69xwp"]
Sep 09 20:35:35.004207 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:35.004175    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="496a7626-e8a3-4ade-9138-ae2123dfd840" path="/var/lib/kubelet/pods/496a7626-e8a3-4ade-9138-ae2123dfd840/volumes"
Sep 09 20:35:35.004597 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:35.004583    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e50b7107-d192-4cdc-8358-0b7dd4096bd1" path="/var/lib/kubelet/pods/e50b7107-d192-4cdc-8358-0b7dd4096bd1/volumes"
Sep 09 20:35:38.454236 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454202    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"]
Sep 09 20:35:38.454647 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454595    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab24bbf6-3f46-45bd-9d9c-d08d024e5079" containerName="payload-pre-processing"
Sep 09 20:35:38.454647 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454612    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab24bbf6-3f46-45bd-9d9c-d08d024e5079" containerName="payload-pre-processing"
Sep 09 20:35:38.454647 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454623    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50b7107-d192-4cdc-8358-0b7dd4096bd1" containerName="main"
Sep 09 20:35:38.454647 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454629    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50b7107-d192-4cdc-8358-0b7dd4096bd1" containerName="main"
Sep 09 20:35:38.454647 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454641    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c683389c-4524-41e6-9501-a649e694a6d6" containerName="payload-processing"
Sep 09 20:35:38.454647 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454647    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c683389c-4524-41e6-9501-a649e694a6d6" containerName="payload-processing"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454654    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b75e187-2af4-4d21-bf81-25e196757820" containerName="revoke-keys"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454659    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b75e187-2af4-4d21-bf81-25e196757820" containerName="revoke-keys"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454667    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9b9f9f-aa25-4398-ae65-abf3a644f103" containerName="payload-pre-processing"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454673    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9b9f9f-aa25-4398-ae65-abf3a644f103" containerName="payload-pre-processing"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454687    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496a7626-e8a3-4ade-9138-ae2123dfd840" containerName="main"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454695    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="496a7626-e8a3-4ade-9138-ae2123dfd840" containerName="main"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454782    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e50b7107-d192-4cdc-8358-0b7dd4096bd1" containerName="main"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454790    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c683389c-4524-41e6-9501-a649e694a6d6" containerName="payload-processing"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454795    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b75e187-2af4-4d21-bf81-25e196757820" containerName="revoke-keys"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454800    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9b9f9f-aa25-4398-ae65-abf3a644f103" containerName="payload-pre-processing"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454809    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab24bbf6-3f46-45bd-9d9c-d08d024e5079" containerName="payload-pre-processing"
Sep 09 20:35:38.454860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.454815    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="496a7626-e8a3-4ade-9138-ae2123dfd840" containerName="main"
Sep 09 20:35:38.459403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.459379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"
Sep 09 20:35:38.461707 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.461679    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:35:38.461837 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.461811    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:35:38.466129 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.466102    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"]
Sep 09 20:35:38.597313 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.597277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e25e65c-936e-46e5-a214-567123233e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv\" (UID: \"9e25e65c-936e-46e5-a214-567123233e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"
Sep 09 20:35:38.698257 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.698225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e25e65c-936e-46e5-a214-567123233e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv\" (UID: \"9e25e65c-936e-46e5-a214-567123233e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"
Sep 09 20:35:38.698865 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.698846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e25e65c-936e-46e5-a214-567123233e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv\" (UID: \"9e25e65c-936e-46e5-a214-567123233e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"
Sep 09 20:35:38.771371 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:38.771253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"
Sep 09 20:35:39.053167 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:39.053140    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"]
Sep 09 20:35:39.054483 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:35:39.054456    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e25e65c_936e_46e5_a214_567123233e45.slice/crio-347fbf03e7eda0717da5bab0d1a26f04e6d8706c5e13775c6a6bb06b0bb100db WatchSource:0}: Error finding container 347fbf03e7eda0717da5bab0d1a26f04e6d8706c5e13775c6a6bb06b0bb100db: Status 404 returned error can't find the container with id 347fbf03e7eda0717da5bab0d1a26f04e6d8706c5e13775c6a6bb06b0bb100db
Sep 09 20:35:39.086903 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:39.086874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv" event={"ID":"9e25e65c-936e-46e5-a214-567123233e45","Type":"ContainerStarted","Data":"347fbf03e7eda0717da5bab0d1a26f04e6d8706c5e13775c6a6bb06b0bb100db"}
Sep 09 20:35:40.094701 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:40.094608    2581 generic.go:356] "Generic (PLEG): container finished" podID="9e25e65c-936e-46e5-a214-567123233e45" containerID="79ab5d00ce2e08a300bef5c45f79ff738d0c21ba6115790d533e01292f8a650d" exitCode=0
Sep 09 20:35:40.094701 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:40.094691    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv" event={"ID":"9e25e65c-936e-46e5-a214-567123233e45","Type":"ContainerDied","Data":"79ab5d00ce2e08a300bef5c45f79ff738d0c21ba6115790d533e01292f8a650d"}
Sep 09 20:35:41.228524 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:41.228498    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"
Sep 09 20:35:41.322150 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:41.322116    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e25e65c-936e-46e5-a214-567123233e45-maas-api-service-ca\") pod \"9e25e65c-936e-46e5-a214-567123233e45\" (UID: \"9e25e65c-936e-46e5-a214-567123233e45\") "
Sep 09 20:35:41.322550 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:41.322520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e25e65c-936e-46e5-a214-567123233e45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e25e65c-936e-46e5-a214-567123233e45" (UID: "9e25e65c-936e-46e5-a214-567123233e45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:41.423134 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:41.423054    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e25e65c-936e-46e5-a214-567123233e45-maas-api-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:42.107605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:42.107570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv" event={"ID":"9e25e65c-936e-46e5-a214-567123233e45","Type":"ContainerDied","Data":"347fbf03e7eda0717da5bab0d1a26f04e6d8706c5e13775c6a6bb06b0bb100db"}
Sep 09 20:35:42.107605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:42.107592    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"
Sep 09 20:35:42.107827 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:42.107623    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="347fbf03e7eda0717da5bab0d1a26f04e6d8706c5e13775c6a6bb06b0bb100db"
Sep 09 20:35:49.624863 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.624827    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"]
Sep 09 20:35:49.625479 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.625413    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e25e65c-936e-46e5-a214-567123233e45" containerName="revoke-keys"
Sep 09 20:35:49.625479 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.625432    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e25e65c-936e-46e5-a214-567123233e45" containerName="revoke-keys"
Sep 09 20:35:49.625619 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.625547    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e25e65c-936e-46e5-a214-567123233e45" containerName="revoke-keys"
Sep 09 20:35:49.628307 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.628287    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:35:49.630835 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.630815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-47e11e-dockercfg-bg7qh\""
Sep 09 20:35:49.630961 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.630871    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-47e11e\""
Sep 09 20:35:49.639162 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.639135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"]
Sep 09 20:35:49.678541 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.678509    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"]
Sep 09 20:35:49.681954 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.681938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:35:49.691663 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.691638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"]
Sep 09 20:35:49.800420 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.800387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2edc2a78-16ee-4f29-9969-2c70f26e969a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k\" (UID: \"2edc2a78-16ee-4f29-9969-2c70f26e969a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:35:49.800581 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.800510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8wxt\" (UniqueName: \"kubernetes.io/projected/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-kube-api-access-z8wxt\") pod \"payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw\" (UID: \"aaf81c7f-b3eb-4631-9568-f841e18a7d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:35:49.800581 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.800544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvz47\" (UniqueName: \"kubernetes.io/projected/2edc2a78-16ee-4f29-9969-2c70f26e969a-kube-api-access-nvz47\") pod \"payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k\" (UID: \"2edc2a78-16ee-4f29-9969-2c70f26e969a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:35:49.800581 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.800563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw\" (UID: \"aaf81c7f-b3eb-4631-9568-f841e18a7d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:35:49.902054 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.901954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw\" (UID: \"aaf81c7f-b3eb-4631-9568-f841e18a7d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:35:49.902054 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.902020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2edc2a78-16ee-4f29-9969-2c70f26e969a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k\" (UID: \"2edc2a78-16ee-4f29-9969-2c70f26e969a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:35:49.902299 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.902136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8wxt\" (UniqueName: \"kubernetes.io/projected/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-kube-api-access-z8wxt\") pod \"payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw\" (UID: \"aaf81c7f-b3eb-4631-9568-f841e18a7d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:35:49.902299 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.902171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvz47\" (UniqueName: \"kubernetes.io/projected/2edc2a78-16ee-4f29-9969-2c70f26e969a-kube-api-access-nvz47\") pod \"payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k\" (UID: \"2edc2a78-16ee-4f29-9969-2c70f26e969a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:35:49.902727 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.902707    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw\" (UID: \"aaf81c7f-b3eb-4631-9568-f841e18a7d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:35:49.902771 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.902734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2edc2a78-16ee-4f29-9969-2c70f26e969a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k\" (UID: \"2edc2a78-16ee-4f29-9969-2c70f26e969a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:35:49.910916 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.910895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvz47\" (UniqueName: \"kubernetes.io/projected/2edc2a78-16ee-4f29-9969-2c70f26e969a-kube-api-access-nvz47\") pod \"payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k\" (UID: \"2edc2a78-16ee-4f29-9969-2c70f26e969a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:35:49.911043 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.910974    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8wxt\" (UniqueName: \"kubernetes.io/projected/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-kube-api-access-z8wxt\") pod \"payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw\" (UID: \"aaf81c7f-b3eb-4631-9568-f841e18a7d9a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:35:49.939982 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.939953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:35:49.994238 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:49.993987    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:35:50.230252 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:50.230229    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"]
Sep 09 20:35:50.231650 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:35:50.231623    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edc2a78_16ee_4f29_9969_2c70f26e969a.slice/crio-dd58f4a98e9a969a2ae9ee665efae1aa134c156eef24144ea74ee52245e5a7a2 WatchSource:0}: Error finding container dd58f4a98e9a969a2ae9ee665efae1aa134c156eef24144ea74ee52245e5a7a2: Status 404 returned error can't find the container with id dd58f4a98e9a969a2ae9ee665efae1aa134c156eef24144ea74ee52245e5a7a2
Sep 09 20:35:50.290404 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:50.288814    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"]
Sep 09 20:35:51.162016 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.161974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k" event={"ID":"2edc2a78-16ee-4f29-9969-2c70f26e969a","Type":"ContainerStarted","Data":"a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c"}
Sep 09 20:35:51.162546 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.162052    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:35:51.162546 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.162080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k" event={"ID":"2edc2a78-16ee-4f29-9969-2c70f26e969a","Type":"ContainerStarted","Data":"dd58f4a98e9a969a2ae9ee665efae1aa134c156eef24144ea74ee52245e5a7a2"}
Sep 09 20:35:51.163771 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.163741    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw" event={"ID":"aaf81c7f-b3eb-4631-9568-f841e18a7d9a","Type":"ContainerStarted","Data":"03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938"}
Sep 09 20:35:51.163926 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.163792    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw" event={"ID":"aaf81c7f-b3eb-4631-9568-f841e18a7d9a","Type":"ContainerStarted","Data":"7230d0f61389240b5fdd4e38dc48b320cbdf2ffa8cec78916320bdb40199af63"}
Sep 09 20:35:51.163926 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.163821    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:35:51.179268 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.179219    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k" podStartSLOduration=2.179202006 podStartE2EDuration="2.179202006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:51.178005584 +0000 UTC m=+1172.671216560" watchObservedRunningTime="2026-09-09 20:35:51.179202006 +0000 UTC m=+1172.672412991"
Sep 09 20:35:51.196050 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.195993    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw" podStartSLOduration=2.195979582 podStartE2EDuration="2.195979582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:51.193309339 +0000 UTC m=+1172.686520325" watchObservedRunningTime="2026-09-09 20:35:51.195979582 +0000 UTC m=+1172.689190566"
Sep 09 20:35:51.667361 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.667307    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"]
Sep 09 20:35:51.667608 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.667580    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd" podUID="484ab58f-2800-4679-be53-38c319a335f5" containerName="payload-processing" containerID="cri-o://95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e" gracePeriod=30
Sep 09 20:35:51.819428 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.819393    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"]
Sep 09 20:35:51.821661 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.821632    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4defd2-vnbsv"]
Sep 09 20:35:51.935824 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:51.935795    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:35:52.021046 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.021010    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vl25\" (UniqueName: \"kubernetes.io/projected/484ab58f-2800-4679-be53-38c319a335f5-kube-api-access-6vl25\") pod \"484ab58f-2800-4679-be53-38c319a335f5\" (UID: \"484ab58f-2800-4679-be53-38c319a335f5\") "
Sep 09 20:35:52.021046 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.021043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/484ab58f-2800-4679-be53-38c319a335f5-plugins-config-volume\") pod \"484ab58f-2800-4679-be53-38c319a335f5\" (UID: \"484ab58f-2800-4679-be53-38c319a335f5\") "
Sep 09 20:35:52.021496 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.021475    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484ab58f-2800-4679-be53-38c319a335f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "484ab58f-2800-4679-be53-38c319a335f5" (UID: "484ab58f-2800-4679-be53-38c319a335f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:35:52.023230 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.023210    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484ab58f-2800-4679-be53-38c319a335f5-kube-api-access-6vl25" (OuterVolumeSpecName: "kube-api-access-6vl25") pod "484ab58f-2800-4679-be53-38c319a335f5" (UID: "484ab58f-2800-4679-be53-38c319a335f5"). InnerVolumeSpecName "kube-api-access-6vl25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:35:52.122090 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.122048    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vl25\" (UniqueName: \"kubernetes.io/projected/484ab58f-2800-4679-be53-38c319a335f5-kube-api-access-6vl25\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:52.122090 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.122082    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/484ab58f-2800-4679-be53-38c319a335f5-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:35:52.171960 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.171870    2581 generic.go:356] "Generic (PLEG): container finished" podID="484ab58f-2800-4679-be53-38c319a335f5" containerID="95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e" exitCode=0
Sep 09 20:35:52.171960 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.171924    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"
Sep 09 20:35:52.172533 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.171961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd" event={"ID":"484ab58f-2800-4679-be53-38c319a335f5","Type":"ContainerDied","Data":"95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e"}
Sep 09 20:35:52.172533 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.172018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd" event={"ID":"484ab58f-2800-4679-be53-38c319a335f5","Type":"ContainerDied","Data":"48b3dd02f6145cb4a4d9ecf283d7834ae6eeab9d5db6538804ebc9a8c4345cc0"}
Sep 09 20:35:52.172533 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.172046    2581 scope.go:117] "RemoveContainer" containerID="95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e"
Sep 09 20:35:52.182265 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.182247    2581 scope.go:117] "RemoveContainer" containerID="95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e"
Sep 09 20:35:52.182598 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:35:52.182582    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e\": container with ID starting with 95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e not found: ID does not exist" containerID="95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e"
Sep 09 20:35:52.182657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.182605    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e"} err="failed to get container status \"95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e\": rpc error: code = NotFound desc = could not find container \"95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e\": container with ID starting with 95e3e97994e91589826e9339f154733b4be06373029b3f01d108ec18f1d47d6e not found: ID does not exist"
Sep 09 20:35:52.194443 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.194403    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"]
Sep 09 20:35:52.198727 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:52.198685    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4defd2-6c85c758d5-4thdd"]
Sep 09 20:35:53.002263 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:53.002225    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484ab58f-2800-4679-be53-38c319a335f5" path="/var/lib/kubelet/pods/484ab58f-2800-4679-be53-38c319a335f5/volumes"
Sep 09 20:35:53.002580 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:35:53.002567    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e25e65c-936e-46e5-a214-567123233e45" path="/var/lib/kubelet/pods/9e25e65c-936e-46e5-a214-567123233e45/volumes"
Sep 09 20:36:02.174016 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:02.173987    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:36:02.174912 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:02.174891    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:36:05.335233 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.335190    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"]
Sep 09 20:36:05.335774 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.335753    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484ab58f-2800-4679-be53-38c319a335f5" containerName="payload-processing"
Sep 09 20:36:05.335823 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.335778    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="484ab58f-2800-4679-be53-38c319a335f5" containerName="payload-processing"
Sep 09 20:36:05.335896 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.335883    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="484ab58f-2800-4679-be53-38c319a335f5" containerName="payload-processing"
Sep 09 20:36:05.340241 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.340216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:36:05.340808 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.340779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f\" (UID: \"1bd19932-cb24-4c3e-9e4f-9d956d76a89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:36:05.340931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.340825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrnk\" (UniqueName: \"kubernetes.io/projected/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-kube-api-access-hhrnk\") pod \"payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f\" (UID: \"1bd19932-cb24-4c3e-9e4f-9d956d76a89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:36:05.343492 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.343468    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-166b94\""
Sep 09 20:36:05.343634 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.343607    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-166b94-dockercfg-mltqj\""
Sep 09 20:36:05.351808 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.351779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"]
Sep 09 20:36:05.441546 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.441503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f\" (UID: \"1bd19932-cb24-4c3e-9e4f-9d956d76a89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:36:05.441734 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.441562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrnk\" (UniqueName: \"kubernetes.io/projected/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-kube-api-access-hhrnk\") pod \"payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f\" (UID: \"1bd19932-cb24-4c3e-9e4f-9d956d76a89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:36:05.442266 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.442242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f\" (UID: \"1bd19932-cb24-4c3e-9e4f-9d956d76a89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:36:05.450150 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.450114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrnk\" (UniqueName: \"kubernetes.io/projected/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-kube-api-access-hhrnk\") pod \"payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f\" (UID: \"1bd19932-cb24-4c3e-9e4f-9d956d76a89d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:36:05.651712 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.651634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:36:05.924398 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:05.924366    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"]
Sep 09 20:36:05.925871 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:36:05.925846    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd19932_cb24_4c3e_9e4f_9d956d76a89d.slice/crio-c10a71d4e561fbf6c4eaad41d1c14af83ff71b5debfe09f443c7303c7c74fe2b WatchSource:0}: Error finding container c10a71d4e561fbf6c4eaad41d1c14af83ff71b5debfe09f443c7303c7c74fe2b: Status 404 returned error can't find the container with id c10a71d4e561fbf6c4eaad41d1c14af83ff71b5debfe09f443c7303c7c74fe2b
Sep 09 20:36:06.263864 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:06.263826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f" event={"ID":"1bd19932-cb24-4c3e-9e4f-9d956d76a89d","Type":"ContainerStarted","Data":"ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c"}
Sep 09 20:36:06.264066 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:06.263911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f" event={"ID":"1bd19932-cb24-4c3e-9e4f-9d956d76a89d","Type":"ContainerStarted","Data":"c10a71d4e561fbf6c4eaad41d1c14af83ff71b5debfe09f443c7303c7c74fe2b"}
Sep 09 20:36:06.264139 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:06.264068    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:36:06.282134 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:06.282065    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f" podStartSLOduration=1.282048761 podStartE2EDuration="1.282048761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:06.278652883 +0000 UTC m=+1187.771863868" watchObservedRunningTime="2026-09-09 20:36:06.282048761 +0000 UTC m=+1187.775259744"
Sep 09 20:36:11.367796 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:11.367763    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7577-1ac19c"]
Sep 09 20:36:11.371299 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:11.371280    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-1ac19c"
Sep 09 20:36:11.373897 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:11.373873    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5fctx\""
Sep 09 20:36:11.381900 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:11.381862    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7577-1ac19c"]
Sep 09 20:36:11.394601 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:11.394575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4zl\" (UniqueName: \"kubernetes.io/projected/c5244de9-5c28-43d8-a38d-3bff5721c85d-kube-api-access-zp4zl\") pod \"test-curl-7577-1ac19c\" (UID: \"c5244de9-5c28-43d8-a38d-3bff5721c85d\") " pod="opendatahub/test-curl-7577-1ac19c"
Sep 09 20:36:11.495735 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:11.495699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4zl\" (UniqueName: \"kubernetes.io/projected/c5244de9-5c28-43d8-a38d-3bff5721c85d-kube-api-access-zp4zl\") pod \"test-curl-7577-1ac19c\" (UID: \"c5244de9-5c28-43d8-a38d-3bff5721c85d\") " pod="opendatahub/test-curl-7577-1ac19c"
Sep 09 20:36:11.506320 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:11.506290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4zl\" (UniqueName: \"kubernetes.io/projected/c5244de9-5c28-43d8-a38d-3bff5721c85d-kube-api-access-zp4zl\") pod \"test-curl-7577-1ac19c\" (UID: \"c5244de9-5c28-43d8-a38d-3bff5721c85d\") " pod="opendatahub/test-curl-7577-1ac19c"
Sep 09 20:36:11.683304 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:11.683273    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-1ac19c"
Sep 09 20:36:11.951631 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:11.951606    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7577-1ac19c"]
Sep 09 20:36:11.953025 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:36:11.952991    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5244de9_5c28_43d8_a38d_3bff5721c85d.slice/crio-745bfd70fe260660575cc668ae987719e7425a90fd88154404f75ec4d5c1ab26 WatchSource:0}: Error finding container 745bfd70fe260660575cc668ae987719e7425a90fd88154404f75ec4d5c1ab26: Status 404 returned error can't find the container with id 745bfd70fe260660575cc668ae987719e7425a90fd88154404f75ec4d5c1ab26
Sep 09 20:36:12.300696 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:12.300604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7577-1ac19c" event={"ID":"c5244de9-5c28-43d8-a38d-3bff5721c85d","Type":"ContainerStarted","Data":"4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989"}
Sep 09 20:36:12.300696 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:12.300653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7577-1ac19c" event={"ID":"c5244de9-5c28-43d8-a38d-3bff5721c85d","Type":"ContainerStarted","Data":"745bfd70fe260660575cc668ae987719e7425a90fd88154404f75ec4d5c1ab26"}
Sep 09 20:36:12.315504 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:12.315460    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7577-1ac19c" podStartSLOduration=1.315445945 podStartE2EDuration="1.315445945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:12.314182858 +0000 UTC m=+1193.807393837" watchObservedRunningTime="2026-09-09 20:36:12.315445945 +0000 UTC m=+1193.808656933"
Sep 09 20:36:13.022329 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.022242    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7577-1ac19c"]
Sep 09 20:36:13.026555 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.026526    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7577-1ac19c"]
Sep 09 20:36:13.162717 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.162684    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7577-3b6fee"]
Sep 09 20:36:13.163070 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.163057    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5244de9-5c28-43d8-a38d-3bff5721c85d" containerName="test-curl-7577-1ac19c"
Sep 09 20:36:13.163127 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.163072    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5244de9-5c28-43d8-a38d-3bff5721c85d" containerName="test-curl-7577-1ac19c"
Sep 09 20:36:13.163163 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.163142    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5244de9-5c28-43d8-a38d-3bff5721c85d" containerName="test-curl-7577-1ac19c"
Sep 09 20:36:13.166265 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.166249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-3b6fee"
Sep 09 20:36:13.173038 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.173009    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7577-3b6fee"]
Sep 09 20:36:13.212895 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.212855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8czs\" (UniqueName: \"kubernetes.io/projected/18bbcb16-328a-4fb4-962b-cb2ec8bb8e93-kube-api-access-t8czs\") pod \"test-curl-7577-3b6fee\" (UID: \"18bbcb16-328a-4fb4-962b-cb2ec8bb8e93\") " pod="opendatahub/test-curl-7577-3b6fee"
Sep 09 20:36:13.306829 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:36:13.306730    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7577-1ac19c" container="test-curl-7577-1ac19c"
Sep 09 20:36:13.313802 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.313765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8czs\" (UniqueName: \"kubernetes.io/projected/18bbcb16-328a-4fb4-962b-cb2ec8bb8e93-kube-api-access-t8czs\") pod \"test-curl-7577-3b6fee\" (UID: \"18bbcb16-328a-4fb4-962b-cb2ec8bb8e93\") " pod="opendatahub/test-curl-7577-3b6fee"
Sep 09 20:36:13.322678 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.322658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8czs\" (UniqueName: \"kubernetes.io/projected/18bbcb16-328a-4fb4-962b-cb2ec8bb8e93-kube-api-access-t8czs\") pod \"test-curl-7577-3b6fee\" (UID: \"18bbcb16-328a-4fb4-962b-cb2ec8bb8e93\") " pod="opendatahub/test-curl-7577-3b6fee"
Sep 09 20:36:13.477453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.477416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-3b6fee"
Sep 09 20:36:13.749498 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:13.749474    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7577-3b6fee"]
Sep 09 20:36:13.750937 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:36:13.750898    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18bbcb16_328a_4fb4_962b_cb2ec8bb8e93.slice/crio-2c952723cdde91770c607194cf2fde49f57a2e3bdb9d92e4aef4eedafccb06c9 WatchSource:0}: Error finding container 2c952723cdde91770c607194cf2fde49f57a2e3bdb9d92e4aef4eedafccb06c9: Status 404 returned error can't find the container with id 2c952723cdde91770c607194cf2fde49f57a2e3bdb9d92e4aef4eedafccb06c9
Sep 09 20:36:14.313914 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:14.313881    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7577-3b6fee" event={"ID":"18bbcb16-328a-4fb4-962b-cb2ec8bb8e93","Type":"ContainerStarted","Data":"c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e"}
Sep 09 20:36:14.314316 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:14.313928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7577-3b6fee" event={"ID":"18bbcb16-328a-4fb4-962b-cb2ec8bb8e93","Type":"ContainerStarted","Data":"2c952723cdde91770c607194cf2fde49f57a2e3bdb9d92e4aef4eedafccb06c9"}
Sep 09 20:36:14.314316 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:14.314073    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7577-1ac19c" podUID="c5244de9-5c28-43d8-a38d-3bff5721c85d" containerName="test-curl-7577-1ac19c" containerID="cri-o://4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989" gracePeriod=2
Sep 09 20:36:14.327774 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:36:14.327741    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7577-1ac19c\" is forbidden: User \"system:node:ip-10-0-135-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-122.ec2.internal' and this object" podUID="c5244de9-5c28-43d8-a38d-3bff5721c85d" pod="opendatahub/test-curl-7577-1ac19c"
Sep 09 20:36:14.969206 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:14.969145    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7577-3b6fee" podStartSLOduration=1.969128048 podStartE2EDuration="1.969128048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:14.325721127 +0000 UTC m=+1195.818932113" watchObservedRunningTime="2026-09-09 20:36:14.969128048 +0000 UTC m=+1196.462339085"
Sep 09 20:36:14.969832 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:14.969810    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7577-3b6fee"]
Sep 09 20:36:14.974055 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:14.974033    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7577-3b6fee"]
Sep 09 20:36:15.320284 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:36:15.320247    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7577-3b6fee" container="test-curl-7577-3b6fee"
Sep 09 20:36:16.327272 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:16.327230    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7577-3b6fee" podUID="18bbcb16-328a-4fb4-962b-cb2ec8bb8e93" containerName="test-curl-7577-3b6fee" containerID="cri-o://c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e" gracePeriod=2
Sep 09 20:36:16.490184 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:16.490160    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-1ac19c"
Sep 09 20:36:16.542435 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:16.542411    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp4zl\" (UniqueName: \"kubernetes.io/projected/c5244de9-5c28-43d8-a38d-3bff5721c85d-kube-api-access-zp4zl\") pod \"c5244de9-5c28-43d8-a38d-3bff5721c85d\" (UID: \"c5244de9-5c28-43d8-a38d-3bff5721c85d\") "
Sep 09 20:36:16.544562 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:16.544537    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5244de9-5c28-43d8-a38d-3bff5721c85d-kube-api-access-zp4zl" (OuterVolumeSpecName: "kube-api-access-zp4zl") pod "c5244de9-5c28-43d8-a38d-3bff5721c85d" (UID: "c5244de9-5c28-43d8-a38d-3bff5721c85d"). InnerVolumeSpecName "kube-api-access-zp4zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:16.643783 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:16.643713    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4zl\" (UniqueName: \"kubernetes.io/projected/c5244de9-5c28-43d8-a38d-3bff5721c85d-kube-api-access-zp4zl\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:36:17.002012 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:17.001974    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5244de9-5c28-43d8-a38d-3bff5721c85d" path="/var/lib/kubelet/pods/c5244de9-5c28-43d8-a38d-3bff5721c85d/volumes"
Sep 09 20:36:17.272956 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:17.272875    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:36:17.333793 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:17.333754    2581 generic.go:356] "Generic (PLEG): container finished" podID="c5244de9-5c28-43d8-a38d-3bff5721c85d" containerID="4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989" exitCode=137
Sep 09 20:36:17.334239 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:17.333799    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-1ac19c"
Sep 09 20:36:17.334239 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:17.333853    2581 scope.go:117] "RemoveContainer" containerID="4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989"
Sep 09 20:36:17.344908 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:17.344892    2581 scope.go:117] "RemoveContainer" containerID="4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989"
Sep 09 20:36:17.345218 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:36:17.345192    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989\": container with ID starting with 4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989 not found: ID does not exist" containerID="4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989"
Sep 09 20:36:17.345316 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:17.345228    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989"} err="failed to get container status \"4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989\": rpc error: code = NotFound desc = could not find container \"4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989\": container with ID starting with 4578c4fac5e3505bae45d9b60fa128c98fbdd8269ccd6c5ff9d53fd5b0c47989 not found: ID does not exist"
Sep 09 20:36:18.508353 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:18.508313    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-3b6fee"
Sep 09 20:36:18.561644 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:18.561605    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8czs\" (UniqueName: \"kubernetes.io/projected/18bbcb16-328a-4fb4-962b-cb2ec8bb8e93-kube-api-access-t8czs\") pod \"18bbcb16-328a-4fb4-962b-cb2ec8bb8e93\" (UID: \"18bbcb16-328a-4fb4-962b-cb2ec8bb8e93\") "
Sep 09 20:36:18.563777 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:18.563751    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bbcb16-328a-4fb4-962b-cb2ec8bb8e93-kube-api-access-t8czs" (OuterVolumeSpecName: "kube-api-access-t8czs") pod "18bbcb16-328a-4fb4-962b-cb2ec8bb8e93" (UID: "18bbcb16-328a-4fb4-962b-cb2ec8bb8e93"). InnerVolumeSpecName "kube-api-access-t8czs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:18.662410 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:18.662315    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8czs\" (UniqueName: \"kubernetes.io/projected/18bbcb16-328a-4fb4-962b-cb2ec8bb8e93-kube-api-access-t8czs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:36:19.003718 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:19.003686    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18bbcb16-328a-4fb4-962b-cb2ec8bb8e93" path="/var/lib/kubelet/pods/18bbcb16-328a-4fb4-962b-cb2ec8bb8e93/volumes"
Sep 09 20:36:19.026989 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:19.026967    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:36:19.031920 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:19.031896    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:36:19.141832 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:19.141802    2581 scope.go:117] "RemoveContainer" containerID="5613878637debb84fbe8d9d3eeb7d87311a533b4b9c5ff49a67885356d2382a8"
Sep 09 20:36:19.346698 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:19.346606    2581 generic.go:356] "Generic (PLEG): container finished" podID="18bbcb16-328a-4fb4-962b-cb2ec8bb8e93" containerID="c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e" exitCode=137
Sep 09 20:36:19.346698 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:19.346649    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7577-3b6fee"
Sep 09 20:36:19.346698 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:19.346660    2581 scope.go:117] "RemoveContainer" containerID="c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e"
Sep 09 20:36:19.355223 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:19.355204    2581 scope.go:117] "RemoveContainer" containerID="c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e"
Sep 09 20:36:19.355526 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:36:19.355508    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e\": container with ID starting with c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e not found: ID does not exist" containerID="c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e"
Sep 09 20:36:19.355598 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:19.355532    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e"} err="failed to get container status \"c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e\": rpc error: code = NotFound desc = could not find container \"c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e\": container with ID starting with c289b5f54023c9e124c86abe2c81abd8145268ce04555be71d5fd83fe0f5301e not found: ID does not exist"
Sep 09 20:36:23.207445 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.207414    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"]
Sep 09 20:36:23.207896 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.207811    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18bbcb16-328a-4fb4-962b-cb2ec8bb8e93" containerName="test-curl-7577-3b6fee"
Sep 09 20:36:23.207896 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.207822    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="18bbcb16-328a-4fb4-962b-cb2ec8bb8e93" containerName="test-curl-7577-3b6fee"
Sep 09 20:36:23.207896 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.207881    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="18bbcb16-328a-4fb4-962b-cb2ec8bb8e93" containerName="test-curl-7577-3b6fee"
Sep 09 20:36:23.212632 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.212613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:36:23.215441 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.215421    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-57f5cbb5\""
Sep 09 20:36:23.222746 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.222722    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"]
Sep 09 20:36:23.237062 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.237038    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"]
Sep 09 20:36:23.241180 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.241164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:36:23.248007 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.247988    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-57f5cbb5-dockercfg-4jh5v\""
Sep 09 20:36:23.253938 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.253919    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"]
Sep 09 20:36:23.302017 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.301986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c86f1c-716d-4379-845e-becbcb3e2f68-plugins-config-volume\") pod \"payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r\" (UID: \"98c86f1c-716d-4379-845e-becbcb3e2f68\") " pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:36:23.302017 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.302019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47lt9\" (UniqueName: \"kubernetes.io/projected/98c86f1c-716d-4379-845e-becbcb3e2f68-kube-api-access-47lt9\") pod \"payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r\" (UID: \"98c86f1c-716d-4379-845e-becbcb3e2f68\") " pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:36:23.302191 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.302058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c\" (UID: \"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:36:23.302191 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.302112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltvf\" (UniqueName: \"kubernetes.io/projected/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-kube-api-access-dltvf\") pod \"payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c\" (UID: \"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:36:23.403052 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.403009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c86f1c-716d-4379-845e-becbcb3e2f68-plugins-config-volume\") pod \"payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r\" (UID: \"98c86f1c-716d-4379-845e-becbcb3e2f68\") " pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:36:23.403052 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.403051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47lt9\" (UniqueName: \"kubernetes.io/projected/98c86f1c-716d-4379-845e-becbcb3e2f68-kube-api-access-47lt9\") pod \"payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r\" (UID: \"98c86f1c-716d-4379-845e-becbcb3e2f68\") " pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:36:23.403321 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.403168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c\" (UID: \"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:36:23.403321 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.403213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltvf\" (UniqueName: \"kubernetes.io/projected/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-kube-api-access-dltvf\") pod \"payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c\" (UID: \"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:36:23.403829 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.403806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c86f1c-716d-4379-845e-becbcb3e2f68-plugins-config-volume\") pod \"payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r\" (UID: \"98c86f1c-716d-4379-845e-becbcb3e2f68\") " pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:36:23.403924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.403832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c\" (UID: \"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:36:23.410961 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.410941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47lt9\" (UniqueName: \"kubernetes.io/projected/98c86f1c-716d-4379-845e-becbcb3e2f68-kube-api-access-47lt9\") pod \"payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r\" (UID: \"98c86f1c-716d-4379-845e-becbcb3e2f68\") " pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:36:23.411247 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.411227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltvf\" (UniqueName: \"kubernetes.io/projected/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-kube-api-access-dltvf\") pod \"payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c\" (UID: \"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:36:23.525316 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.525226    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:36:23.550460 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.550431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:36:23.799810 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.799787    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"]
Sep 09 20:36:23.801430 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:36:23.801396    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a3dbaf_d681_41bd_b1d2_2daefde00c9b.slice/crio-8e3f4c5e48b28ea292b7cfd200003af1ff2228a74e83571d78f47561d3beb761 WatchSource:0}: Error finding container 8e3f4c5e48b28ea292b7cfd200003af1ff2228a74e83571d78f47561d3beb761: Status 404 returned error can't find the container with id 8e3f4c5e48b28ea292b7cfd200003af1ff2228a74e83571d78f47561d3beb761
Sep 09 20:36:23.836086 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:23.836063    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"]
Sep 09 20:36:23.837318 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:36:23.837295    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c86f1c_716d_4379_845e_becbcb3e2f68.slice/crio-70229f7d62561fd5c74428a19f0bbf82b9c7e1a0f11c14b1da2cf37d3c370690 WatchSource:0}: Error finding container 70229f7d62561fd5c74428a19f0bbf82b9c7e1a0f11c14b1da2cf37d3c370690: Status 404 returned error can't find the container with id 70229f7d62561fd5c74428a19f0bbf82b9c7e1a0f11c14b1da2cf37d3c370690
Sep 09 20:36:24.377973 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:24.377938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c" event={"ID":"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b","Type":"ContainerStarted","Data":"f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737"}
Sep 09 20:36:24.378481 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:24.378002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c" event={"ID":"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b","Type":"ContainerStarted","Data":"8e3f4c5e48b28ea292b7cfd200003af1ff2228a74e83571d78f47561d3beb761"}
Sep 09 20:36:24.378481 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:24.378120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:36:24.379761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:24.379732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r" event={"ID":"98c86f1c-716d-4379-845e-becbcb3e2f68","Type":"ContainerStarted","Data":"a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a"}
Sep 09 20:36:24.379859 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:24.379782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r" event={"ID":"98c86f1c-716d-4379-845e-becbcb3e2f68","Type":"ContainerStarted","Data":"70229f7d62561fd5c74428a19f0bbf82b9c7e1a0f11c14b1da2cf37d3c370690"}
Sep 09 20:36:24.379859 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:24.379816    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:36:24.424182 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:24.424137    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c" podStartSLOduration=1.4240969159999999 podStartE2EDuration="1.424096916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:24.422495376 +0000 UTC m=+1205.915706359" watchObservedRunningTime="2026-09-09 20:36:24.424096916 +0000 UTC m=+1205.917307899"
Sep 09 20:36:24.458313 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:24.458268    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r" podStartSLOduration=1.458253685 podStartE2EDuration="1.458253685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:24.455822451 +0000 UTC m=+1205.949033435" watchObservedRunningTime="2026-09-09 20:36:24.458253685 +0000 UTC m=+1205.951464669"
Sep 09 20:36:35.388582 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:35.388552    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:36:35.389179 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:35.389159    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:36:35.988584 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:35.988549    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"]
Sep 09 20:36:35.992508 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:35.992487    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:36:35.995151 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:35.995132    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4101f2\""
Sep 09 20:36:35.995249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:35.995172    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4101f2-dockercfg-5rwlh\""
Sep 09 20:36:36.002640 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.002619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"]
Sep 09 20:36:36.019677 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.019655    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"]
Sep 09 20:36:36.022971 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.022956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:36:36.032011 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.031991    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"]
Sep 09 20:36:36.120997 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.120961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae35504-f913-4961-8764-b19dd9c9e7e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x\" (UID: \"5ae35504-f913-4961-8764-b19dd9c9e7e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:36:36.121183 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.121034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc92021-806a-46b8-bdb2-8a5b40431f02-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp\" (UID: \"0cc92021-806a-46b8-bdb2-8a5b40431f02\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:36:36.121183 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.121078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vkgv\" (UniqueName: \"kubernetes.io/projected/0cc92021-806a-46b8-bdb2-8a5b40431f02-kube-api-access-4vkgv\") pod \"payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp\" (UID: \"0cc92021-806a-46b8-bdb2-8a5b40431f02\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:36:36.121183 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.121158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzszv\" (UniqueName: \"kubernetes.io/projected/5ae35504-f913-4961-8764-b19dd9c9e7e9-kube-api-access-nzszv\") pod \"payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x\" (UID: \"5ae35504-f913-4961-8764-b19dd9c9e7e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:36:36.222187 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.222154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzszv\" (UniqueName: \"kubernetes.io/projected/5ae35504-f913-4961-8764-b19dd9c9e7e9-kube-api-access-nzszv\") pod \"payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x\" (UID: \"5ae35504-f913-4961-8764-b19dd9c9e7e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:36:36.222421 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.222205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae35504-f913-4961-8764-b19dd9c9e7e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x\" (UID: \"5ae35504-f913-4961-8764-b19dd9c9e7e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:36:36.222421 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.222248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc92021-806a-46b8-bdb2-8a5b40431f02-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp\" (UID: \"0cc92021-806a-46b8-bdb2-8a5b40431f02\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:36:36.222421 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.222276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vkgv\" (UniqueName: \"kubernetes.io/projected/0cc92021-806a-46b8-bdb2-8a5b40431f02-kube-api-access-4vkgv\") pod \"payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp\" (UID: \"0cc92021-806a-46b8-bdb2-8a5b40431f02\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:36:36.222966 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.222945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae35504-f913-4961-8764-b19dd9c9e7e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x\" (UID: \"5ae35504-f913-4961-8764-b19dd9c9e7e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:36:36.223013 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.222976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc92021-806a-46b8-bdb2-8a5b40431f02-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp\" (UID: \"0cc92021-806a-46b8-bdb2-8a5b40431f02\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:36:36.230674 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.230652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vkgv\" (UniqueName: \"kubernetes.io/projected/0cc92021-806a-46b8-bdb2-8a5b40431f02-kube-api-access-4vkgv\") pod \"payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp\" (UID: \"0cc92021-806a-46b8-bdb2-8a5b40431f02\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:36:36.230779 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.230653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzszv\" (UniqueName: \"kubernetes.io/projected/5ae35504-f913-4961-8764-b19dd9c9e7e9-kube-api-access-nzszv\") pod \"payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x\" (UID: \"5ae35504-f913-4961-8764-b19dd9c9e7e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:36:36.304478 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.304396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:36:36.334579 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.334551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:36:36.581019 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.580991    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"]
Sep 09 20:36:36.582536 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:36:36.582507    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc92021_806a_46b8_bdb2_8a5b40431f02.slice/crio-e9625d14dfbf01ff48dda8248db62524c85da963fec5bcc7d99fa85bb63dc767 WatchSource:0}: Error finding container e9625d14dfbf01ff48dda8248db62524c85da963fec5bcc7d99fa85bb63dc767: Status 404 returned error can't find the container with id e9625d14dfbf01ff48dda8248db62524c85da963fec5bcc7d99fa85bb63dc767
Sep 09 20:36:36.626668 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:36.626642    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"]
Sep 09 20:36:37.468194 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:37.468152    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x" event={"ID":"5ae35504-f913-4961-8764-b19dd9c9e7e9","Type":"ContainerStarted","Data":"ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4"}
Sep 09 20:36:37.468419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:37.468215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x" event={"ID":"5ae35504-f913-4961-8764-b19dd9c9e7e9","Type":"ContainerStarted","Data":"bd01dd3069e48885bec3eaa9ef98be86e2230c5e052ad1402d7ced8d9441e8df"}
Sep 09 20:36:37.468419 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:37.468310    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:36:37.469968 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:37.469941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp" event={"ID":"0cc92021-806a-46b8-bdb2-8a5b40431f02","Type":"ContainerStarted","Data":"c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c"}
Sep 09 20:36:37.470135 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:37.469984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp" event={"ID":"0cc92021-806a-46b8-bdb2-8a5b40431f02","Type":"ContainerStarted","Data":"e9625d14dfbf01ff48dda8248db62524c85da963fec5bcc7d99fa85bb63dc767"}
Sep 09 20:36:37.470135 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:37.470004    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:36:37.485512 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:37.485466    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x" podStartSLOduration=1.485452677 podStartE2EDuration="1.485452677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:37.483472604 +0000 UTC m=+1218.976683589" watchObservedRunningTime="2026-09-09 20:36:37.485452677 +0000 UTC m=+1218.978663661"
Sep 09 20:36:37.500600 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:37.500557    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp" podStartSLOduration=2.50054272 podStartE2EDuration="2.50054272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:37.498277761 +0000 UTC m=+1218.991488741" watchObservedRunningTime="2026-09-09 20:36:37.50054272 +0000 UTC m=+1218.993753720"
Sep 09 20:36:39.930756 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:39.930716    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"]
Sep 09 20:36:39.934405 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:39.934385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:39.936940 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:39.936914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-57f5cbb5\"/\"openshift-service-ca.crt\""
Sep 09 20:36:39.938011 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:39.937972    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-57f5cbb5\"/\"e2e-lifecycle-model-57f5cbb5-kserve-self-signed-certs\""
Sep 09 20:36:39.938117 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:39.938020    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-57f5cbb5\"/\"default-dockercfg-pppvr\""
Sep 09 20:36:39.938188 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:39.938110    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-57f5cbb5\"/\"kube-root-ca.crt\""
Sep 09 20:36:39.942710 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:39.942685    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"]
Sep 09 20:36:40.056184 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.056151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-home\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.056417 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.056190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/751391d9-7f50-4d58-bcd1-bdcbafd86497-tls-certs\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.056417 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.056324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-tmp-dir\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.056417 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.056404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-model-cache\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.056575 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.056437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7xh\" (UniqueName: \"kubernetes.io/projected/751391d9-7f50-4d58-bcd1-bdcbafd86497-kube-api-access-2c7xh\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.056575 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.056502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-dshm\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.157087 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.157054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-home\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.157087 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.157099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/751391d9-7f50-4d58-bcd1-bdcbafd86497-tls-certs\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.157405 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.157169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-tmp-dir\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.157405 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.157191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-model-cache\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.157405 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.157218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7xh\" (UniqueName: \"kubernetes.io/projected/751391d9-7f50-4d58-bcd1-bdcbafd86497-kube-api-access-2c7xh\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.157405 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.157274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-dshm\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.157632 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.157559    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-home\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.157632 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.157581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-tmp-dir\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.157705 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.157667    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-model-cache\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.159582 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.159559    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-dshm\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.159800 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.159783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/751391d9-7f50-4d58-bcd1-bdcbafd86497-tls-certs\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.164994 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.164976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7xh\" (UniqueName: \"kubernetes.io/projected/751391d9-7f50-4d58-bcd1-bdcbafd86497-kube-api-access-2c7xh\") pod \"e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") " pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.246180 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.246100    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:40.521119 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:40.521079    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"]
Sep 09 20:36:40.522422 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:36:40.522396    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod751391d9_7f50_4d58_bcd1_bdcbafd86497.slice/crio-e8220dad5de2c0035201ebcda8636de9be55d0926b68106fbb36804574aa8658 WatchSource:0}: Error finding container e8220dad5de2c0035201ebcda8636de9be55d0926b68106fbb36804574aa8658: Status 404 returned error can't find the container with id e8220dad5de2c0035201ebcda8636de9be55d0926b68106fbb36804574aa8658
Sep 09 20:36:41.495110 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:41.495069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j" event={"ID":"751391d9-7f50-4d58-bcd1-bdcbafd86497","Type":"ContainerStarted","Data":"e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a"}
Sep 09 20:36:41.495110 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:41.495115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j" event={"ID":"751391d9-7f50-4d58-bcd1-bdcbafd86497","Type":"ContainerStarted","Data":"e8220dad5de2c0035201ebcda8636de9be55d0926b68106fbb36804574aa8658"}
Sep 09 20:36:41.515010 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:41.514959    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j" podStartSLOduration=2.514944601 podStartE2EDuration="2.514944601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:36:41.511887675 +0000 UTC m=+1223.005098659" watchObservedRunningTime="2026-09-09 20:36:41.514944601 +0000 UTC m=+1223.008155585"
Sep 09 20:36:48.479531 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:48.479489    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:36:48.480017 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:48.479570    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:36:50.246427 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:50.246393    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:50.246844 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:50.246449    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:50.259074 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:50.259049    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:36:50.563517 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:36:50.563426    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:37:08.823892 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:08.823854    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"]
Sep 09 20:37:08.827537 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:08.827520    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"
Sep 09 20:37:08.830418 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:08.830387    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:37:08.830576 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:08.830549    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:37:08.834932 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:08.834910    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"]
Sep 09 20:37:08.912588 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:08.912554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e473ec3f-b29d-4315-8077-c3ca15e8e6cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k\" (UID: \"e473ec3f-b29d-4315-8077-c3ca15e8e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"
Sep 09 20:37:09.013547 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.013512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e473ec3f-b29d-4315-8077-c3ca15e8e6cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k\" (UID: \"e473ec3f-b29d-4315-8077-c3ca15e8e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"
Sep 09 20:37:09.014099 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.014079    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e473ec3f-b29d-4315-8077-c3ca15e8e6cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k\" (UID: \"e473ec3f-b29d-4315-8077-c3ca15e8e6cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"
Sep 09 20:37:09.135264 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.135188    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"]
Sep 09 20:37:09.138852 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.138834    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.139832 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.139811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"
Sep 09 20:37:09.142911 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.142816    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"kube-root-ca.crt\""
Sep 09 20:37:09.142911 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.142832    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"openshift-service-ca.crt\""
Sep 09 20:37:09.142911 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.142816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"auth-test-model-166b94-kserve-self-signed-certs\""
Sep 09 20:37:09.143079 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.142843    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-166b94\"/\"default-dockercfg-9zkh8\""
Sep 09 20:37:09.152391 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.152325    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"]
Sep 09 20:37:09.215801 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.215770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-home\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.215935 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.215837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfltq\" (UniqueName: \"kubernetes.io/projected/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-kube-api-access-nfltq\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.215935 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.215871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tmp-dir\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.215935 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.215897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tls-certs\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.216044 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.215956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-dshm\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.216044 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.215992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-model-cache\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.316546 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.316507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-home\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.316743 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.316573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfltq\" (UniqueName: \"kubernetes.io/projected/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-kube-api-access-nfltq\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.316743 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.316602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tmp-dir\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.316743 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.316632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tls-certs\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.316743 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.316662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-dshm\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.316992 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.316763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-model-cache\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.316992 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.316957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-home\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.317116 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.317020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tmp-dir\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.317198 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.317176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-model-cache\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.319450 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.319429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-dshm\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.319815 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.319793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tls-certs\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.325190 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.325167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfltq\" (UniqueName: \"kubernetes.io/projected/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-kube-api-access-nfltq\") pod \"auth-test-model-166b94-kserve-95698db67-dmc5j\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") " pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.414791 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.414764    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"]
Sep 09 20:37:09.416134 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:37:09.416106    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode473ec3f_b29d_4315_8077_c3ca15e8e6cb.slice/crio-8e79fb237551d8e8a4567f139f50ad0ea499e46e5cb8f547938ec149ce8f84ec WatchSource:0}: Error finding container 8e79fb237551d8e8a4567f139f50ad0ea499e46e5cb8f547938ec149ce8f84ec: Status 404 returned error can't find the container with id 8e79fb237551d8e8a4567f139f50ad0ea499e46e5cb8f547938ec149ce8f84ec
Sep 09 20:37:09.448740 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.448718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:09.672274 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.672207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k" event={"ID":"e473ec3f-b29d-4315-8077-c3ca15e8e6cb","Type":"ContainerStarted","Data":"27392997fdac519d5e6842f80824fe4cb25061495f5f14f0499f54530032c924"}
Sep 09 20:37:09.672274 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.672261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k" event={"ID":"e473ec3f-b29d-4315-8077-c3ca15e8e6cb","Type":"ContainerStarted","Data":"8e79fb237551d8e8a4567f139f50ad0ea499e46e5cb8f547938ec149ce8f84ec"}
Sep 09 20:37:09.690594 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.690524    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k" podStartSLOduration=1.690508723 podStartE2EDuration="1.690508723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:09.689455819 +0000 UTC m=+1251.182666798" watchObservedRunningTime="2026-09-09 20:37:09.690508723 +0000 UTC m=+1251.183719707"
Sep 09 20:37:09.724726 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:09.724701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"]
Sep 09 20:37:09.725584 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:37:09.725560    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9393cf39_9209_4f0b_9fa5_12a7f7ea514d.slice/crio-9afd0b3bcfc8e27774ad9161cf48438b3599405ffad44e237dac4e4de07c73e3 WatchSource:0}: Error finding container 9afd0b3bcfc8e27774ad9161cf48438b3599405ffad44e237dac4e4de07c73e3: Status 404 returned error can't find the container with id 9afd0b3bcfc8e27774ad9161cf48438b3599405ffad44e237dac4e4de07c73e3
Sep 09 20:37:10.679581 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:10.679546    2581 generic.go:356] "Generic (PLEG): container finished" podID="e473ec3f-b29d-4315-8077-c3ca15e8e6cb" containerID="27392997fdac519d5e6842f80824fe4cb25061495f5f14f0499f54530032c924" exitCode=0
Sep 09 20:37:10.680087 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:10.679596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k" event={"ID":"e473ec3f-b29d-4315-8077-c3ca15e8e6cb","Type":"ContainerDied","Data":"27392997fdac519d5e6842f80824fe4cb25061495f5f14f0499f54530032c924"}
Sep 09 20:37:10.681241 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:10.681219    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j" event={"ID":"9393cf39-9209-4f0b-9fa5-12a7f7ea514d","Type":"ContainerStarted","Data":"f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6"}
Sep 09 20:37:10.681392 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:10.681247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j" event={"ID":"9393cf39-9209-4f0b-9fa5-12a7f7ea514d","Type":"ContainerStarted","Data":"9afd0b3bcfc8e27774ad9161cf48438b3599405ffad44e237dac4e4de07c73e3"}
Sep 09 20:37:10.724414 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:10.724369    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j" podStartSLOduration=1.724356172 podStartE2EDuration="1.724356172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:10.723773315 +0000 UTC m=+1252.216984301" watchObservedRunningTime="2026-09-09 20:37:10.724356172 +0000 UTC m=+1252.217567147"
Sep 09 20:37:11.826238 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:11.826215    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"
Sep 09 20:37:11.943639 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:11.943542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e473ec3f-b29d-4315-8077-c3ca15e8e6cb-maas-api-service-ca\") pod \"e473ec3f-b29d-4315-8077-c3ca15e8e6cb\" (UID: \"e473ec3f-b29d-4315-8077-c3ca15e8e6cb\") "
Sep 09 20:37:11.943977 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:11.943949    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e473ec3f-b29d-4315-8077-c3ca15e8e6cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e473ec3f-b29d-4315-8077-c3ca15e8e6cb" (UID: "e473ec3f-b29d-4315-8077-c3ca15e8e6cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:12.045072 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:12.045039    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e473ec3f-b29d-4315-8077-c3ca15e8e6cb-maas-api-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:37:12.694207 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:12.694178    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"
Sep 09 20:37:12.694388 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:12.694212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k" event={"ID":"e473ec3f-b29d-4315-8077-c3ca15e8e6cb","Type":"ContainerDied","Data":"8e79fb237551d8e8a4567f139f50ad0ea499e46e5cb8f547938ec149ce8f84ec"}
Sep 09 20:37:12.694388 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:12.694265    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e79fb237551d8e8a4567f139f50ad0ea499e46e5cb8f547938ec149ce8f84ec"
Sep 09 20:37:19.188763 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:19.188725    2581 scope.go:117] "RemoveContainer" containerID="9421c8b52ffe0425b443e3a9d6b259910c8413445e95366c1c02d97aacc2da7d"
Sep 09 20:37:19.449092 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:19.448987    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:19.449092 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:19.449033    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:19.462110 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:19.462080    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:19.748731 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:19.748643    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:37:24.760240 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:24.760204    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"]
Sep 09 20:37:24.760734 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:24.760478    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r" podUID="98c86f1c-716d-4379-845e-becbcb3e2f68" containerName="payload-processing" containerID="cri-o://a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a" gracePeriod=30
Sep 09 20:37:24.770648 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:24.770619    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"]
Sep 09 20:37:24.770931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:24.770883    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c" podUID="f9a3dbaf-d681-41bd-b1d2-2daefde00c9b" containerName="payload-pre-processing" containerID="cri-o://f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737" gracePeriod=30
Sep 09 20:37:24.933124 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:24.932736    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"]
Sep 09 20:37:24.936430 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:24.936335    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57f5cbb5-jjn4k"]
Sep 09 20:37:25.004959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.004918    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e473ec3f-b29d-4315-8077-c3ca15e8e6cb" path="/var/lib/kubelet/pods/e473ec3f-b29d-4315-8077-c3ca15e8e6cb/volumes"
Sep 09 20:37:25.054160 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.054136    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:37:25.056641 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.056622    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:37:25.176045 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.176014    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-plugins-config-volume\") pod \"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b\" (UID: \"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b\") "
Sep 09 20:37:25.176045 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.176051    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dltvf\" (UniqueName: \"kubernetes.io/projected/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-kube-api-access-dltvf\") pod \"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b\" (UID: \"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b\") "
Sep 09 20:37:25.176325 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.176131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c86f1c-716d-4379-845e-becbcb3e2f68-plugins-config-volume\") pod \"98c86f1c-716d-4379-845e-becbcb3e2f68\" (UID: \"98c86f1c-716d-4379-845e-becbcb3e2f68\") "
Sep 09 20:37:25.176325 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.176157    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47lt9\" (UniqueName: \"kubernetes.io/projected/98c86f1c-716d-4379-845e-becbcb3e2f68-kube-api-access-47lt9\") pod \"98c86f1c-716d-4379-845e-becbcb3e2f68\" (UID: \"98c86f1c-716d-4379-845e-becbcb3e2f68\") "
Sep 09 20:37:25.176491 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.176441    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9a3dbaf-d681-41bd-b1d2-2daefde00c9b" (UID: "f9a3dbaf-d681-41bd-b1d2-2daefde00c9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:25.176551 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.176514    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c86f1c-716d-4379-845e-becbcb3e2f68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98c86f1c-716d-4379-845e-becbcb3e2f68" (UID: "98c86f1c-716d-4379-845e-becbcb3e2f68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:37:25.178538 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.178517    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c86f1c-716d-4379-845e-becbcb3e2f68-kube-api-access-47lt9" (OuterVolumeSpecName: "kube-api-access-47lt9") pod "98c86f1c-716d-4379-845e-becbcb3e2f68" (UID: "98c86f1c-716d-4379-845e-becbcb3e2f68"). InnerVolumeSpecName "kube-api-access-47lt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:25.178629 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.178575    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-kube-api-access-dltvf" (OuterVolumeSpecName: "kube-api-access-dltvf") pod "f9a3dbaf-d681-41bd-b1d2-2daefde00c9b" (UID: "f9a3dbaf-d681-41bd-b1d2-2daefde00c9b"). InnerVolumeSpecName "kube-api-access-dltvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:25.277792 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.277710    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47lt9\" (UniqueName: \"kubernetes.io/projected/98c86f1c-716d-4379-845e-becbcb3e2f68-kube-api-access-47lt9\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:37:25.277792 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.277738    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:37:25.277792 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.277747    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dltvf\" (UniqueName: \"kubernetes.io/projected/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b-kube-api-access-dltvf\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:37:25.277792 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.277756    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c86f1c-716d-4379-845e-becbcb3e2f68-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:37:25.778085 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.778050    2581 generic.go:356] "Generic (PLEG): container finished" podID="98c86f1c-716d-4379-845e-becbcb3e2f68" containerID="a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a" exitCode=0
Sep 09 20:37:25.778566 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.778126    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"
Sep 09 20:37:25.778566 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.778146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r" event={"ID":"98c86f1c-716d-4379-845e-becbcb3e2f68","Type":"ContainerDied","Data":"a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a"}
Sep 09 20:37:25.778566 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.778234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r" event={"ID":"98c86f1c-716d-4379-845e-becbcb3e2f68","Type":"ContainerDied","Data":"70229f7d62561fd5c74428a19f0bbf82b9c7e1a0f11c14b1da2cf37d3c370690"}
Sep 09 20:37:25.778566 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.778262    2581 scope.go:117] "RemoveContainer" containerID="a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a"
Sep 09 20:37:25.780195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.780167    2581 generic.go:356] "Generic (PLEG): container finished" podID="f9a3dbaf-d681-41bd-b1d2-2daefde00c9b" containerID="f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737" exitCode=0
Sep 09 20:37:25.780360 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.780234    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"
Sep 09 20:37:25.780360 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.780256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c" event={"ID":"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b","Type":"ContainerDied","Data":"f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737"}
Sep 09 20:37:25.780360 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.780315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c" event={"ID":"f9a3dbaf-d681-41bd-b1d2-2daefde00c9b","Type":"ContainerDied","Data":"8e3f4c5e48b28ea292b7cfd200003af1ff2228a74e83571d78f47561d3beb761"}
Sep 09 20:37:25.791030 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.791012    2581 scope.go:117] "RemoveContainer" containerID="a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a"
Sep 09 20:37:25.791334 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:37:25.791312    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a\": container with ID starting with a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a not found: ID does not exist" containerID="a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a"
Sep 09 20:37:25.791442 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.791365    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a"} err="failed to get container status \"a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a\": rpc error: code = NotFound desc = could not find container \"a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a\": container with ID starting with a14bb02764b43ac83b71aff3481ae978843dcaa555213c118932ae3f890dc39a not found: ID does not exist"
Sep 09 20:37:25.791442 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.791384    2581 scope.go:117] "RemoveContainer" containerID="f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737"
Sep 09 20:37:25.802633 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.802606    2581 scope.go:117] "RemoveContainer" containerID="f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737"
Sep 09 20:37:25.802950 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:37:25.802925    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737\": container with ID starting with f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737 not found: ID does not exist" containerID="f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737"
Sep 09 20:37:25.803096 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.802959    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737"} err="failed to get container status \"f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737\": rpc error: code = NotFound desc = could not find container \"f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737\": container with ID starting with f022ec42c379f94d9255bf0842b8c8e32ec7c24824456185a7170b4e157bf737 not found: ID does not exist"
Sep 09 20:37:25.804751 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.804730    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"]
Sep 09 20:37:25.808049 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.808030    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57f5cbb5-59cf4bdb69-lln4r"]
Sep 09 20:37:25.811505 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:37:25.811483    2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c86f1c_716d_4379_845e_becbcb3e2f68.slice/crio-70229f7d62561fd5c74428a19f0bbf82b9c7e1a0f11c14b1da2cf37d3c370690\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c86f1c_716d_4379_845e_becbcb3e2f68.slice\": RecentStats: unable to find data in memory cache]"
Sep 09 20:37:25.818011 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.817991    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"]
Sep 09 20:37:25.822030 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:25.822007    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57f5cbb5-74d856fbcd-dcm2c"]
Sep 09 20:37:27.002883 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:27.002847    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98c86f1c-716d-4379-845e-becbcb3e2f68" path="/var/lib/kubelet/pods/98c86f1c-716d-4379-845e-becbcb3e2f68/volumes"
Sep 09 20:37:27.003264 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:27.003185    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9a3dbaf-d681-41bd-b1d2-2daefde00c9b" path="/var/lib/kubelet/pods/f9a3dbaf-d681-41bd-b1d2-2daefde00c9b/volumes"
Sep 09 20:37:33.254538 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:33.254489    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"]
Sep 09 20:37:48.275063 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.275013    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j" podUID="751391d9-7f50-4d58-bcd1-bdcbafd86497" containerName="main" containerID="cri-o://e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a" gracePeriod=45
Sep 09 20:37:48.528878 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.528815    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:37:48.585918 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.585881    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/751391d9-7f50-4d58-bcd1-bdcbafd86497-tls-certs\") pod \"751391d9-7f50-4d58-bcd1-bdcbafd86497\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") "
Sep 09 20:37:48.586074 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.585928    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-dshm\") pod \"751391d9-7f50-4d58-bcd1-bdcbafd86497\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") "
Sep 09 20:37:48.586074 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.585972    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-model-cache\") pod \"751391d9-7f50-4d58-bcd1-bdcbafd86497\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") "
Sep 09 20:37:48.586074 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.586049    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-tmp-dir\") pod \"751391d9-7f50-4d58-bcd1-bdcbafd86497\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") "
Sep 09 20:37:48.586231 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.586083    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2c7xh\" (UniqueName: \"kubernetes.io/projected/751391d9-7f50-4d58-bcd1-bdcbafd86497-kube-api-access-2c7xh\") pod \"751391d9-7f50-4d58-bcd1-bdcbafd86497\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") "
Sep 09 20:37:48.586231 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.586120    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-home\") pod \"751391d9-7f50-4d58-bcd1-bdcbafd86497\" (UID: \"751391d9-7f50-4d58-bcd1-bdcbafd86497\") "
Sep 09 20:37:48.586509 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.586473    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "751391d9-7f50-4d58-bcd1-bdcbafd86497" (UID: "751391d9-7f50-4d58-bcd1-bdcbafd86497"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:48.586509 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.586487    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-home" (OuterVolumeSpecName: "home") pod "751391d9-7f50-4d58-bcd1-bdcbafd86497" (UID: "751391d9-7f50-4d58-bcd1-bdcbafd86497"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:48.586910 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.586565    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-model-cache" (OuterVolumeSpecName: "model-cache") pod "751391d9-7f50-4d58-bcd1-bdcbafd86497" (UID: "751391d9-7f50-4d58-bcd1-bdcbafd86497"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:48.588387 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.588360    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751391d9-7f50-4d58-bcd1-bdcbafd86497-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "751391d9-7f50-4d58-bcd1-bdcbafd86497" (UID: "751391d9-7f50-4d58-bcd1-bdcbafd86497"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:48.588493 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.588409    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751391d9-7f50-4d58-bcd1-bdcbafd86497-kube-api-access-2c7xh" (OuterVolumeSpecName: "kube-api-access-2c7xh") pod "751391d9-7f50-4d58-bcd1-bdcbafd86497" (UID: "751391d9-7f50-4d58-bcd1-bdcbafd86497"). InnerVolumeSpecName "kube-api-access-2c7xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:48.588602 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.588581    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-dshm" (OuterVolumeSpecName: "dshm") pod "751391d9-7f50-4d58-bcd1-bdcbafd86497" (UID: "751391d9-7f50-4d58-bcd1-bdcbafd86497"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:48.687595 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.687543    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-tmp-dir\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:37:48.687595 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.687589    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c7xh\" (UniqueName: \"kubernetes.io/projected/751391d9-7f50-4d58-bcd1-bdcbafd86497-kube-api-access-2c7xh\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:37:48.687595 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.687603    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-home\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:37:48.687862 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.687617    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/751391d9-7f50-4d58-bcd1-bdcbafd86497-tls-certs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:37:48.687862 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.687627    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-dshm\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:37:48.687862 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.687639    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/751391d9-7f50-4d58-bcd1-bdcbafd86497-model-cache\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:37:48.923134 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.923040    2581 generic.go:356] "Generic (PLEG): container finished" podID="751391d9-7f50-4d58-bcd1-bdcbafd86497" containerID="e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a" exitCode=0
Sep 09 20:37:48.923134 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.923085    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"
Sep 09 20:37:48.923134 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.923125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j" event={"ID":"751391d9-7f50-4d58-bcd1-bdcbafd86497","Type":"ContainerDied","Data":"e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a"}
Sep 09 20:37:48.923460 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.923175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j" event={"ID":"751391d9-7f50-4d58-bcd1-bdcbafd86497","Type":"ContainerDied","Data":"e8220dad5de2c0035201ebcda8636de9be55d0926b68106fbb36804574aa8658"}
Sep 09 20:37:48.923460 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.923198    2581 scope.go:117] "RemoveContainer" containerID="e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a"
Sep 09 20:37:48.933024 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.933007    2581 scope.go:117] "RemoveContainer" containerID="e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a"
Sep 09 20:37:48.933332 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:37:48.933314    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a\": container with ID starting with e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a not found: ID does not exist" containerID="e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a"
Sep 09 20:37:48.933418 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.933357    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a"} err="failed to get container status \"e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a\": rpc error: code = NotFound desc = could not find container \"e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a\": container with ID starting with e7edeabcf15f4ece8dd630edd677fba7825da603ec42babf091bed7afd5fd81a not found: ID does not exist"
Sep 09 20:37:48.944533 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.944506    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"]
Sep 09 20:37:48.949930 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:48.949909    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-57f5cbb5/e2e-lifecycle-model-57f5cbb5-kserve-569477475-cb82j"]
Sep 09 20:37:49.002303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:37:49.002265    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="751391d9-7f50-4d58-bcd1-bdcbafd86497" path="/var/lib/kubelet/pods/751391d9-7f50-4d58-bcd1-bdcbafd86497/volumes"
Sep 09 20:38:19.218998 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:38:19.218962    2581 scope.go:117] "RemoveContainer" containerID="f990ea45ef085ecd6c5cae78cc52f0158643bcc1cf3122e3bd8e684fdaf3f53f"
Sep 09 20:39:18.173870 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.173835    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"]
Sep 09 20:39:18.174328 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174210    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c86f1c-716d-4379-845e-becbcb3e2f68" containerName="payload-processing"
Sep 09 20:39:18.174328 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174222    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c86f1c-716d-4379-845e-becbcb3e2f68" containerName="payload-processing"
Sep 09 20:39:18.174328 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174239    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a3dbaf-d681-41bd-b1d2-2daefde00c9b" containerName="payload-pre-processing"
Sep 09 20:39:18.174328 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174245    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a3dbaf-d681-41bd-b1d2-2daefde00c9b" containerName="payload-pre-processing"
Sep 09 20:39:18.174328 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174255    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e473ec3f-b29d-4315-8077-c3ca15e8e6cb" containerName="revoke-keys"
Sep 09 20:39:18.174328 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174261    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e473ec3f-b29d-4315-8077-c3ca15e8e6cb" containerName="revoke-keys"
Sep 09 20:39:18.174328 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174275    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="751391d9-7f50-4d58-bcd1-bdcbafd86497" containerName="main"
Sep 09 20:39:18.174328 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174281    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="751391d9-7f50-4d58-bcd1-bdcbafd86497" containerName="main"
Sep 09 20:39:18.174616 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174368    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e473ec3f-b29d-4315-8077-c3ca15e8e6cb" containerName="revoke-keys"
Sep 09 20:39:18.174616 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174377    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="751391d9-7f50-4d58-bcd1-bdcbafd86497" containerName="main"
Sep 09 20:39:18.174616 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174384    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="98c86f1c-716d-4379-845e-becbcb3e2f68" containerName="payload-processing"
Sep 09 20:39:18.174616 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.174391    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9a3dbaf-d681-41bd-b1d2-2daefde00c9b" containerName="payload-pre-processing"
Sep 09 20:39:18.177428 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.177411    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:39:18.179882 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.179857    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-077e89\""
Sep 09 20:39:18.180017 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.179863    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-077e89-dockercfg-vwld5\""
Sep 09 20:39:18.188819 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.188788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"]
Sep 09 20:39:18.191235 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.191211    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"]
Sep 09 20:39:18.194770 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.194751    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:39:18.204543 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.204515    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"]
Sep 09 20:39:18.222399 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.222372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fe1f38-d201-4e69-aacc-9ed61a460bcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq\" (UID: \"80fe1f38-d201-4e69-aacc-9ed61a460bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:39:18.222521 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.222410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2x4\" (UniqueName: \"kubernetes.io/projected/80fe1f38-d201-4e69-aacc-9ed61a460bcb-kube-api-access-lf2x4\") pod \"payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq\" (UID: \"80fe1f38-d201-4e69-aacc-9ed61a460bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:39:18.222521 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.222445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smc4z\" (UniqueName: \"kubernetes.io/projected/d4607a0f-445c-4a25-af5c-fe0e21e7e666-kube-api-access-smc4z\") pod \"payload-processing-e2e-api-a-077e89-5955d94c99-bthnl\" (UID: \"d4607a0f-445c-4a25-af5c-fe0e21e7e666\") " pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:39:18.222629 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.222605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4607a0f-445c-4a25-af5c-fe0e21e7e666-plugins-config-volume\") pod \"payload-processing-e2e-api-a-077e89-5955d94c99-bthnl\" (UID: \"d4607a0f-445c-4a25-af5c-fe0e21e7e666\") " pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:39:18.324019 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.323980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4607a0f-445c-4a25-af5c-fe0e21e7e666-plugins-config-volume\") pod \"payload-processing-e2e-api-a-077e89-5955d94c99-bthnl\" (UID: \"d4607a0f-445c-4a25-af5c-fe0e21e7e666\") " pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:39:18.324194 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.324031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fe1f38-d201-4e69-aacc-9ed61a460bcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq\" (UID: \"80fe1f38-d201-4e69-aacc-9ed61a460bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:39:18.324194 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.324065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2x4\" (UniqueName: \"kubernetes.io/projected/80fe1f38-d201-4e69-aacc-9ed61a460bcb-kube-api-access-lf2x4\") pod \"payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq\" (UID: \"80fe1f38-d201-4e69-aacc-9ed61a460bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:39:18.324194 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.324119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smc4z\" (UniqueName: \"kubernetes.io/projected/d4607a0f-445c-4a25-af5c-fe0e21e7e666-kube-api-access-smc4z\") pod \"payload-processing-e2e-api-a-077e89-5955d94c99-bthnl\" (UID: \"d4607a0f-445c-4a25-af5c-fe0e21e7e666\") " pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:39:18.324810 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.324789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4607a0f-445c-4a25-af5c-fe0e21e7e666-plugins-config-volume\") pod \"payload-processing-e2e-api-a-077e89-5955d94c99-bthnl\" (UID: \"d4607a0f-445c-4a25-af5c-fe0e21e7e666\") " pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:39:18.324860 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.324847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fe1f38-d201-4e69-aacc-9ed61a460bcb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq\" (UID: \"80fe1f38-d201-4e69-aacc-9ed61a460bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:39:18.332502 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.332473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smc4z\" (UniqueName: \"kubernetes.io/projected/d4607a0f-445c-4a25-af5c-fe0e21e7e666-kube-api-access-smc4z\") pod \"payload-processing-e2e-api-a-077e89-5955d94c99-bthnl\" (UID: \"d4607a0f-445c-4a25-af5c-fe0e21e7e666\") " pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:39:18.332615 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.332478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2x4\" (UniqueName: \"kubernetes.io/projected/80fe1f38-d201-4e69-aacc-9ed61a460bcb-kube-api-access-lf2x4\") pod \"payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq\" (UID: \"80fe1f38-d201-4e69-aacc-9ed61a460bcb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:39:18.489165 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.489127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:39:18.506830 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.506757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:39:18.772390 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.772362    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"]
Sep 09 20:39:18.774688 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:39:18.774656    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fe1f38_d201_4e69_aacc_9ed61a460bcb.slice/crio-e40f15e873c60efa13891a6ffca659e8b523d23d746b1e6847da059aac361ad3 WatchSource:0}: Error finding container e40f15e873c60efa13891a6ffca659e8b523d23d746b1e6847da059aac361ad3: Status 404 returned error can't find the container with id e40f15e873c60efa13891a6ffca659e8b523d23d746b1e6847da059aac361ad3
Sep 09 20:39:18.805226 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:18.805203    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"]
Sep 09 20:39:18.812413 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:39:18.812381    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4607a0f_445c_4a25_af5c_fe0e21e7e666.slice/crio-27e1b4d8ba59b89df4a6d262cf47314d3cfac2471b0c0da45b326d180b33b0b8 WatchSource:0}: Error finding container 27e1b4d8ba59b89df4a6d262cf47314d3cfac2471b0c0da45b326d180b33b0b8: Status 404 returned error can't find the container with id 27e1b4d8ba59b89df4a6d262cf47314d3cfac2471b0c0da45b326d180b33b0b8
Sep 09 20:39:19.441744 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:19.441705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl" event={"ID":"d4607a0f-445c-4a25-af5c-fe0e21e7e666","Type":"ContainerStarted","Data":"1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe"}
Sep 09 20:39:19.442234 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:19.441777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:39:19.442234 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:19.441800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl" event={"ID":"d4607a0f-445c-4a25-af5c-fe0e21e7e666","Type":"ContainerStarted","Data":"27e1b4d8ba59b89df4a6d262cf47314d3cfac2471b0c0da45b326d180b33b0b8"}
Sep 09 20:39:19.443502 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:19.443475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq" event={"ID":"80fe1f38-d201-4e69-aacc-9ed61a460bcb","Type":"ContainerStarted","Data":"43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982"}
Sep 09 20:39:19.443650 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:19.443516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq" event={"ID":"80fe1f38-d201-4e69-aacc-9ed61a460bcb","Type":"ContainerStarted","Data":"e40f15e873c60efa13891a6ffca659e8b523d23d746b1e6847da059aac361ad3"}
Sep 09 20:39:19.443650 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:19.443583    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:39:19.459229 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:19.459181    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl" podStartSLOduration=1.459165334 podStartE2EDuration="1.459165334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:19.458190215 +0000 UTC m=+1380.951401211" watchObservedRunningTime="2026-09-09 20:39:19.459165334 +0000 UTC m=+1380.952376364"
Sep 09 20:39:19.475450 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:19.475397    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq" podStartSLOduration=1.47538139 podStartE2EDuration="1.47538139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:19.472784231 +0000 UTC m=+1380.965995218" watchObservedRunningTime="2026-09-09 20:39:19.47538139 +0000 UTC m=+1380.968592373"
Sep 09 20:39:23.467479 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:23.467440    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"]
Sep 09 20:39:30.451529 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:30.451493    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:39:30.452031 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:30.451580    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:39:38.488043 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.488001    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j" podUID="9393cf39-9209-4f0b-9fa5-12a7f7ea514d" containerName="main" containerID="cri-o://f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6" gracePeriod=45
Sep 09 20:39:38.747381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.747306    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:39:38.916069 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.916033    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tmp-dir\") pod \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") "
Sep 09 20:39:38.916261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.916092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-home\") pod \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") "
Sep 09 20:39:38.916261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.916110    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfltq\" (UniqueName: \"kubernetes.io/projected/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-kube-api-access-nfltq\") pod \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") "
Sep 09 20:39:38.916261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.916128    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tls-certs\") pod \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") "
Sep 09 20:39:38.916261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.916185    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-model-cache\") pod \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") "
Sep 09 20:39:38.916261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.916228    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-dshm\") pod \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\" (UID: \"9393cf39-9209-4f0b-9fa5-12a7f7ea514d\") "
Sep 09 20:39:38.916572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.916371    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9393cf39-9209-4f0b-9fa5-12a7f7ea514d" (UID: "9393cf39-9209-4f0b-9fa5-12a7f7ea514d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:38.916572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.916410    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-home" (OuterVolumeSpecName: "home") pod "9393cf39-9209-4f0b-9fa5-12a7f7ea514d" (UID: "9393cf39-9209-4f0b-9fa5-12a7f7ea514d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:38.916572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.916539    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9393cf39-9209-4f0b-9fa5-12a7f7ea514d" (UID: "9393cf39-9209-4f0b-9fa5-12a7f7ea514d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:38.916572 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.916561    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tmp-dir\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:39:38.916717 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.916578    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-home\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:39:38.918571 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.918543    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-dshm" (OuterVolumeSpecName: "dshm") pod "9393cf39-9209-4f0b-9fa5-12a7f7ea514d" (UID: "9393cf39-9209-4f0b-9fa5-12a7f7ea514d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:39:38.918647 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.918606    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9393cf39-9209-4f0b-9fa5-12a7f7ea514d" (UID: "9393cf39-9209-4f0b-9fa5-12a7f7ea514d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:39:38.918704 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:38.918650    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-kube-api-access-nfltq" (OuterVolumeSpecName: "kube-api-access-nfltq") pod "9393cf39-9209-4f0b-9fa5-12a7f7ea514d" (UID: "9393cf39-9209-4f0b-9fa5-12a7f7ea514d"). InnerVolumeSpecName "kube-api-access-nfltq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:39:39.017492 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.017466    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfltq\" (UniqueName: \"kubernetes.io/projected/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-kube-api-access-nfltq\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:39:39.017492 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.017493    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-tls-certs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:39:39.017661 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.017506    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-model-cache\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:39:39.017661 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.017517    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9393cf39-9209-4f0b-9fa5-12a7f7ea514d-dshm\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:39:39.569284 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.569249    2581 generic.go:356] "Generic (PLEG): container finished" podID="9393cf39-9209-4f0b-9fa5-12a7f7ea514d" containerID="f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6" exitCode=0
Sep 09 20:39:39.569762 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.569307    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"
Sep 09 20:39:39.569762 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.569333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j" event={"ID":"9393cf39-9209-4f0b-9fa5-12a7f7ea514d","Type":"ContainerDied","Data":"f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6"}
Sep 09 20:39:39.569762 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.569400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j" event={"ID":"9393cf39-9209-4f0b-9fa5-12a7f7ea514d","Type":"ContainerDied","Data":"9afd0b3bcfc8e27774ad9161cf48438b3599405ffad44e237dac4e4de07c73e3"}
Sep 09 20:39:39.569762 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.569417    2581 scope.go:117] "RemoveContainer" containerID="f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6"
Sep 09 20:39:39.578768 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.578746    2581 scope.go:117] "RemoveContainer" containerID="f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6"
Sep 09 20:39:39.579042 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:39:39.579023    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6\": container with ID starting with f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6 not found: ID does not exist" containerID="f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6"
Sep 09 20:39:39.579093 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.579049    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6"} err="failed to get container status \"f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6\": rpc error: code = NotFound desc = could not find container \"f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6\": container with ID starting with f424c6416a52270943f78e0a816197bd9dd5cab1f4f6384430fd3c50da0179d6 not found: ID does not exist"
Sep 09 20:39:39.586580 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.586546    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"]
Sep 09 20:39:39.592093 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:39.592065    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-166b94/auth-test-model-166b94-kserve-95698db67-dmc5j"]
Sep 09 20:39:41.003061 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:41.003026    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9393cf39-9209-4f0b-9fa5-12a7f7ea514d" path="/var/lib/kubelet/pods/9393cf39-9209-4f0b-9fa5-12a7f7ea514d/volumes"
Sep 09 20:39:48.501143 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.501090    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"]
Sep 09 20:39:48.501673 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.501657    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9393cf39-9209-4f0b-9fa5-12a7f7ea514d" containerName="main"
Sep 09 20:39:48.501719 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.501677    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9393cf39-9209-4f0b-9fa5-12a7f7ea514d" containerName="main"
Sep 09 20:39:48.501783 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.501773    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9393cf39-9209-4f0b-9fa5-12a7f7ea514d" containerName="main"
Sep 09 20:39:48.505321 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.505299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:39:48.508096 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.508073    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-06f7dc\""
Sep 09 20:39:48.508297 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.508265    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-06f7dc-dockercfg-tkzk8\""
Sep 09 20:39:48.519282 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.519250    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"]
Sep 09 20:39:48.522479 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.522453    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"]
Sep 09 20:39:48.533726 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.533697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:39:48.537200 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.537092    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"]
Sep 09 20:39:48.607118 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.607086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6dtj\" (UniqueName: \"kubernetes.io/projected/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-kube-api-access-k6dtj\") pod \"payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7\" (UID: \"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:39:48.607285 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.607131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7\" (UID: \"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:39:48.708596 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.708563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4818fc-961e-4572-a7aa-c2ca81f41c44-plugins-config-volume\") pod \"payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr\" (UID: \"9a4818fc-961e-4572-a7aa-c2ca81f41c44\") " pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:39:48.708798 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.708621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dtj\" (UniqueName: \"kubernetes.io/projected/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-kube-api-access-k6dtj\") pod \"payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7\" (UID: \"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:39:48.708798 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.708697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7\" (UID: \"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:39:48.708930 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.708818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tkbg\" (UniqueName: \"kubernetes.io/projected/9a4818fc-961e-4572-a7aa-c2ca81f41c44-kube-api-access-8tkbg\") pod \"payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr\" (UID: \"9a4818fc-961e-4572-a7aa-c2ca81f41c44\") " pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:39:48.709446 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.709428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7\" (UID: \"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:39:48.716848 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.716810    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6dtj\" (UniqueName: \"kubernetes.io/projected/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-kube-api-access-k6dtj\") pod \"payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7\" (UID: \"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:39:48.810401 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.810292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4818fc-961e-4572-a7aa-c2ca81f41c44-plugins-config-volume\") pod \"payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr\" (UID: \"9a4818fc-961e-4572-a7aa-c2ca81f41c44\") " pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:39:48.810550 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.810430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tkbg\" (UniqueName: \"kubernetes.io/projected/9a4818fc-961e-4572-a7aa-c2ca81f41c44-kube-api-access-8tkbg\") pod \"payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr\" (UID: \"9a4818fc-961e-4572-a7aa-c2ca81f41c44\") " pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:39:48.811207 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.811183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4818fc-961e-4572-a7aa-c2ca81f41c44-plugins-config-volume\") pod \"payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr\" (UID: \"9a4818fc-961e-4572-a7aa-c2ca81f41c44\") " pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:39:48.817065 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.817040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:39:48.818428 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.818403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tkbg\" (UniqueName: \"kubernetes.io/projected/9a4818fc-961e-4572-a7aa-c2ca81f41c44-kube-api-access-8tkbg\") pod \"payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr\" (UID: \"9a4818fc-961e-4572-a7aa-c2ca81f41c44\") " pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:39:48.848592 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:48.848554    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:39:49.098948 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:49.098921    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"]
Sep 09 20:39:49.100146 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:39:49.100123    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07bb019_6d83_4a99_a32e_8c1a7c4ea57a.slice/crio-a54a33599005fc96c32c7599e72ca7e01bc0be02b78abb94d81290bb5276366d WatchSource:0}: Error finding container a54a33599005fc96c32c7599e72ca7e01bc0be02b78abb94d81290bb5276366d: Status 404 returned error can't find the container with id a54a33599005fc96c32c7599e72ca7e01bc0be02b78abb94d81290bb5276366d
Sep 09 20:39:49.136120 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:49.136097    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"]
Sep 09 20:39:49.138529 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:39:49.138507    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4818fc_961e_4572_a7aa_c2ca81f41c44.slice/crio-60ff49204cfc869182274d613abb83e32524f20b401e81cefe0ebf94dc82cf38 WatchSource:0}: Error finding container 60ff49204cfc869182274d613abb83e32524f20b401e81cefe0ebf94dc82cf38: Status 404 returned error can't find the container with id 60ff49204cfc869182274d613abb83e32524f20b401e81cefe0ebf94dc82cf38
Sep 09 20:39:49.633252 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:49.633211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr" event={"ID":"9a4818fc-961e-4572-a7aa-c2ca81f41c44","Type":"ContainerStarted","Data":"48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c"}
Sep 09 20:39:49.633782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:49.633474    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:39:49.633782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:49.633522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr" event={"ID":"9a4818fc-961e-4572-a7aa-c2ca81f41c44","Type":"ContainerStarted","Data":"60ff49204cfc869182274d613abb83e32524f20b401e81cefe0ebf94dc82cf38"}
Sep 09 20:39:49.635068 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:49.635039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7" event={"ID":"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a","Type":"ContainerStarted","Data":"062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a"}
Sep 09 20:39:49.635189 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:49.635089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7" event={"ID":"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a","Type":"ContainerStarted","Data":"a54a33599005fc96c32c7599e72ca7e01bc0be02b78abb94d81290bb5276366d"}
Sep 09 20:39:49.635189 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:49.635135    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:39:49.653568 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:49.653526    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr" podStartSLOduration=1.653513918 podStartE2EDuration="1.653513918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:49.649931467 +0000 UTC m=+1411.143142481" watchObservedRunningTime="2026-09-09 20:39:49.653513918 +0000 UTC m=+1411.146724946"
Sep 09 20:39:49.666023 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:39:49.665979    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7" podStartSLOduration=1.665968444 podStartE2EDuration="1.665968444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:39:49.664998239 +0000 UTC m=+1411.158209222" watchObservedRunningTime="2026-09-09 20:39:49.665968444 +0000 UTC m=+1411.159179428"
Sep 09 20:40:00.643504 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:00.643472    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:40:00.643911 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:00.643550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:40:16.549168 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:16.549077    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"]
Sep 09 20:40:16.554157 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:16.554137    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"
Sep 09 20:40:16.557431 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:16.557409    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:40:16.557694 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:16.557409    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:40:16.560187 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:16.560163    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"]
Sep 09 20:40:16.635549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:16.635516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-077e89-qjgbj\" (UID: \"93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"
Sep 09 20:40:16.736746 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:16.736713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-077e89-qjgbj\" (UID: \"93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"
Sep 09 20:40:16.737396 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:16.737378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-077e89-qjgbj\" (UID: \"93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"
Sep 09 20:40:16.868084 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:16.867997    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"
Sep 09 20:40:17.140122 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:17.140094    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"]
Sep 09 20:40:17.142540 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:40:17.142505    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93716f6f_3d27_48cf_9eb5_bd9a2ef4dbe7.slice/crio-38eff0b886b98fcf5b7af1fb3e6a7b6f593978ded1dc2ea543ed68f0540feb7a WatchSource:0}: Error finding container 38eff0b886b98fcf5b7af1fb3e6a7b6f593978ded1dc2ea543ed68f0540feb7a: Status 404 returned error can't find the container with id 38eff0b886b98fcf5b7af1fb3e6a7b6f593978ded1dc2ea543ed68f0540feb7a
Sep 09 20:40:17.806470 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:17.806375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj" event={"ID":"93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7","Type":"ContainerStarted","Data":"8c6d4c67b0984a6577e739df1c52933cb87939e387ba6d100b7eed61b6604f16"}
Sep 09 20:40:17.806470 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:17.806437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj" event={"ID":"93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7","Type":"ContainerStarted","Data":"38eff0b886b98fcf5b7af1fb3e6a7b6f593978ded1dc2ea543ed68f0540feb7a"}
Sep 09 20:40:17.821754 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:17.821691    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj" podStartSLOduration=1.821673155 podStartE2EDuration="1.821673155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:17.820276238 +0000 UTC m=+1439.313487222" watchObservedRunningTime="2026-09-09 20:40:17.821673155 +0000 UTC m=+1439.314884138"
Sep 09 20:40:18.051555 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.051524    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"]
Sep 09 20:40:18.054979 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.054957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.057677 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.057618    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"openshift-service-ca.crt\""
Sep 09 20:40:18.058027 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.058010    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"sub-test-model-4101f2-kserve-self-signed-certs\""
Sep 09 20:40:18.058261 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.058236    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"kube-root-ca.crt\""
Sep 09 20:40:18.058516 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.058499    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"default-dockercfg-pmn8m\""
Sep 09 20:40:18.065999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.065978    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"]
Sep 09 20:40:18.149403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.149367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tmp-dir\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.149403 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.149401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmkbn\" (UniqueName: \"kubernetes.io/projected/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-kube-api-access-zmkbn\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.149625 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.149472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tls-certs\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.149625 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.149518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-home\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.149625 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.149563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-dshm\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.149625 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.149577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-model-cache\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.250688 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.250644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-dshm\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.250688 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.250687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-model-cache\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.250947 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.250739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tmp-dir\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.250947 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.250771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmkbn\" (UniqueName: \"kubernetes.io/projected/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-kube-api-access-zmkbn\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.250947 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.250829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tls-certs\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.250947 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.250880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-home\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.251227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.251205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tmp-dir\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.251302 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.251231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-model-cache\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.251369 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.251326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-home\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.253150 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.253124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-dshm\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.253406 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.253390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tls-certs\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.267297 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.267266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmkbn\" (UniqueName: \"kubernetes.io/projected/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-kube-api-access-zmkbn\") pod \"sub-test-model-4101f2-kserve-859db84bc9-wsv8r\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.365979 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.365890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:18.641048 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.641022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"]
Sep 09 20:40:18.641900 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:40:18.641881    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0a74beb_4f4d_4916_bc7d_3630ab9997ef.slice/crio-a0e0b817e991507deaeb69049e8b81cc262c391304dba3304441adcfd25285a7 WatchSource:0}: Error finding container a0e0b817e991507deaeb69049e8b81cc262c391304dba3304441adcfd25285a7: Status 404 returned error can't find the container with id a0e0b817e991507deaeb69049e8b81cc262c391304dba3304441adcfd25285a7
Sep 09 20:40:18.813507 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.813471    2581 generic.go:356] "Generic (PLEG): container finished" podID="93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7" containerID="8c6d4c67b0984a6577e739df1c52933cb87939e387ba6d100b7eed61b6604f16" exitCode=0
Sep 09 20:40:18.813947 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.813556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj" event={"ID":"93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7","Type":"ContainerDied","Data":"8c6d4c67b0984a6577e739df1c52933cb87939e387ba6d100b7eed61b6604f16"}
Sep 09 20:40:18.815203 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.815181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r" event={"ID":"a0a74beb-4f4d-4916-bc7d-3630ab9997ef","Type":"ContainerStarted","Data":"845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a"}
Sep 09 20:40:18.815322 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.815211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r" event={"ID":"a0a74beb-4f4d-4916-bc7d-3630ab9997ef","Type":"ContainerStarted","Data":"a0e0b817e991507deaeb69049e8b81cc262c391304dba3304441adcfd25285a7"}
Sep 09 20:40:18.846517 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:18.846461    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r" podStartSLOduration=0.846446092 podStartE2EDuration="846.446092ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:18.8438889 +0000 UTC m=+1440.337099894" watchObservedRunningTime="2026-09-09 20:40:18.846446092 +0000 UTC m=+1440.339657074"
Sep 09 20:40:19.957973 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:19.957945    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"
Sep 09 20:40:20.063959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:20.063927    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7-maas-api-service-ca\") pod \"93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7\" (UID: \"93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7\") "
Sep 09 20:40:20.064483 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:20.064454    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7" (UID: "93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:20.165641 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:20.165548    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7-maas-api-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:40:20.828363 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:20.828298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj" event={"ID":"93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7","Type":"ContainerDied","Data":"38eff0b886b98fcf5b7af1fb3e6a7b6f593978ded1dc2ea543ed68f0540feb7a"}
Sep 09 20:40:20.828539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:20.828411    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38eff0b886b98fcf5b7af1fb3e6a7b6f593978ded1dc2ea543ed68f0540feb7a"
Sep 09 20:40:20.828539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:20.828311    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"
Sep 09 20:40:28.366722 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:28.366674    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:28.367124 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:28.366736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:28.378967 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:28.378941    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:28.885801 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:28.885772    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:40:32.062211 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.062177    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"]
Sep 09 20:40:32.062992 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.062962    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl" podUID="d4607a0f-445c-4a25-af5c-fe0e21e7e666" containerName="payload-processing" containerID="cri-o://1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe" gracePeriod=30
Sep 09 20:40:32.066939 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.066915    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"]
Sep 09 20:40:32.067244 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.067179    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq" podUID="80fe1f38-d201-4e69-aacc-9ed61a460bcb" containerName="payload-pre-processing" containerID="cri-o://43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982" gracePeriod=30
Sep 09 20:40:32.214606 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.214555    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"]
Sep 09 20:40:32.217136 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.217099    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-077e89-qjgbj"]
Sep 09 20:40:32.329696 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.329673    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:40:32.354292 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.354268    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:40:32.490577 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.490545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smc4z\" (UniqueName: \"kubernetes.io/projected/d4607a0f-445c-4a25-af5c-fe0e21e7e666-kube-api-access-smc4z\") pod \"d4607a0f-445c-4a25-af5c-fe0e21e7e666\" (UID: \"d4607a0f-445c-4a25-af5c-fe0e21e7e666\") "
Sep 09 20:40:32.490787 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.490595    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fe1f38-d201-4e69-aacc-9ed61a460bcb-plugins-config-volume\") pod \"80fe1f38-d201-4e69-aacc-9ed61a460bcb\" (UID: \"80fe1f38-d201-4e69-aacc-9ed61a460bcb\") "
Sep 09 20:40:32.490787 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.490648    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf2x4\" (UniqueName: \"kubernetes.io/projected/80fe1f38-d201-4e69-aacc-9ed61a460bcb-kube-api-access-lf2x4\") pod \"80fe1f38-d201-4e69-aacc-9ed61a460bcb\" (UID: \"80fe1f38-d201-4e69-aacc-9ed61a460bcb\") "
Sep 09 20:40:32.490787 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.490665    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4607a0f-445c-4a25-af5c-fe0e21e7e666-plugins-config-volume\") pod \"d4607a0f-445c-4a25-af5c-fe0e21e7e666\" (UID: \"d4607a0f-445c-4a25-af5c-fe0e21e7e666\") "
Sep 09 20:40:32.491054 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.491028    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4607a0f-445c-4a25-af5c-fe0e21e7e666-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4607a0f-445c-4a25-af5c-fe0e21e7e666" (UID: "d4607a0f-445c-4a25-af5c-fe0e21e7e666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:32.491054 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.491037    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fe1f38-d201-4e69-aacc-9ed61a460bcb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80fe1f38-d201-4e69-aacc-9ed61a460bcb" (UID: "80fe1f38-d201-4e69-aacc-9ed61a460bcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:32.492877 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.492853    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fe1f38-d201-4e69-aacc-9ed61a460bcb-kube-api-access-lf2x4" (OuterVolumeSpecName: "kube-api-access-lf2x4") pod "80fe1f38-d201-4e69-aacc-9ed61a460bcb" (UID: "80fe1f38-d201-4e69-aacc-9ed61a460bcb"). InnerVolumeSpecName "kube-api-access-lf2x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:32.492980 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.492910    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4607a0f-445c-4a25-af5c-fe0e21e7e666-kube-api-access-smc4z" (OuterVolumeSpecName: "kube-api-access-smc4z") pod "d4607a0f-445c-4a25-af5c-fe0e21e7e666" (UID: "d4607a0f-445c-4a25-af5c-fe0e21e7e666"). InnerVolumeSpecName "kube-api-access-smc4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:32.592106 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.592013    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smc4z\" (UniqueName: \"kubernetes.io/projected/d4607a0f-445c-4a25-af5c-fe0e21e7e666-kube-api-access-smc4z\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.592106 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.592054    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80fe1f38-d201-4e69-aacc-9ed61a460bcb-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.592106 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.592063    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf2x4\" (UniqueName: \"kubernetes.io/projected/80fe1f38-d201-4e69-aacc-9ed61a460bcb-kube-api-access-lf2x4\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.592106 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.592072    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4607a0f-445c-4a25-af5c-fe0e21e7e666-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:40:32.899438 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.899323    2581 generic.go:356] "Generic (PLEG): container finished" podID="80fe1f38-d201-4e69-aacc-9ed61a460bcb" containerID="43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982" exitCode=0
Sep 09 20:40:32.899438 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.899388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq" event={"ID":"80fe1f38-d201-4e69-aacc-9ed61a460bcb","Type":"ContainerDied","Data":"43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982"}
Sep 09 20:40:32.899661 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.899454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq" event={"ID":"80fe1f38-d201-4e69-aacc-9ed61a460bcb","Type":"ContainerDied","Data":"e40f15e873c60efa13891a6ffca659e8b523d23d746b1e6847da059aac361ad3"}
Sep 09 20:40:32.899661 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.899477    2581 scope.go:117] "RemoveContainer" containerID="43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982"
Sep 09 20:40:32.899661 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.899398    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"
Sep 09 20:40:32.901042 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.901019    2581 generic.go:356] "Generic (PLEG): container finished" podID="d4607a0f-445c-4a25-af5c-fe0e21e7e666" containerID="1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe" exitCode=0
Sep 09 20:40:32.901168 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.901049    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"
Sep 09 20:40:32.901168 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.901083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl" event={"ID":"d4607a0f-445c-4a25-af5c-fe0e21e7e666","Type":"ContainerDied","Data":"1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe"}
Sep 09 20:40:32.901168 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.901132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl" event={"ID":"d4607a0f-445c-4a25-af5c-fe0e21e7e666","Type":"ContainerDied","Data":"27e1b4d8ba59b89df4a6d262cf47314d3cfac2471b0c0da45b326d180b33b0b8"}
Sep 09 20:40:32.910076 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.910055    2581 scope.go:117] "RemoveContainer" containerID="43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982"
Sep 09 20:40:32.910410 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:40:32.910381    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982\": container with ID starting with 43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982 not found: ID does not exist" containerID="43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982"
Sep 09 20:40:32.910520 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.910423    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982"} err="failed to get container status \"43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982\": rpc error: code = NotFound desc = could not find container \"43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982\": container with ID starting with 43993fe0bcc6f69c3a1553f66d8385709d2ccf6b694b620001a9446969cec982 not found: ID does not exist"
Sep 09 20:40:32.910520 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.910445    2581 scope.go:117] "RemoveContainer" containerID="1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe"
Sep 09 20:40:32.918711 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.918694    2581 scope.go:117] "RemoveContainer" containerID="1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe"
Sep 09 20:40:32.918964 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:40:32.918945    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe\": container with ID starting with 1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe not found: ID does not exist" containerID="1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe"
Sep 09 20:40:32.919050 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.918969    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe"} err="failed to get container status \"1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe\": rpc error: code = NotFound desc = could not find container \"1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe\": container with ID starting with 1e80ac67a6e59d420e8fc4669e005fbb435ab36111a077c3e28f67610e5017fe not found: ID does not exist"
Sep 09 20:40:32.923864 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.923843    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"]
Sep 09 20:40:32.931998 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.931978    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-077e89-6dbf65bc86-55phq"]
Sep 09 20:40:32.941992 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.941966    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"]
Sep 09 20:40:32.945659 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:32.945639    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-077e89-5955d94c99-bthnl"]
Sep 09 20:40:33.002620 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:33.002590    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80fe1f38-d201-4e69-aacc-9ed61a460bcb" path="/var/lib/kubelet/pods/80fe1f38-d201-4e69-aacc-9ed61a460bcb/volumes"
Sep 09 20:40:33.002959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:33.002945    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7" path="/var/lib/kubelet/pods/93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7/volumes"
Sep 09 20:40:33.003292 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:33.003280    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4607a0f-445c-4a25-af5c-fe0e21e7e666" path="/var/lib/kubelet/pods/d4607a0f-445c-4a25-af5c-fe0e21e7e666/volumes"
Sep 09 20:40:39.051053 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.051019    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"]
Sep 09 20:40:39.051453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.051417    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4607a0f-445c-4a25-af5c-fe0e21e7e666" containerName="payload-processing"
Sep 09 20:40:39.051453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.051429    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4607a0f-445c-4a25-af5c-fe0e21e7e666" containerName="payload-processing"
Sep 09 20:40:39.051453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.051443    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80fe1f38-d201-4e69-aacc-9ed61a460bcb" containerName="payload-pre-processing"
Sep 09 20:40:39.051453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.051448    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="80fe1f38-d201-4e69-aacc-9ed61a460bcb" containerName="payload-pre-processing"
Sep 09 20:40:39.051586 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.051459    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7" containerName="revoke-keys"
Sep 09 20:40:39.051586 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.051464    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7" containerName="revoke-keys"
Sep 09 20:40:39.051586 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.051518    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="93716f6f-3d27-48cf-9eb5-bd9a2ef4dbe7" containerName="revoke-keys"
Sep 09 20:40:39.051586 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.051529    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="80fe1f38-d201-4e69-aacc-9ed61a460bcb" containerName="payload-pre-processing"
Sep 09 20:40:39.051586 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.051536    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4607a0f-445c-4a25-af5c-fe0e21e7e666" containerName="payload-processing"
Sep 09 20:40:39.056137 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.056120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"
Sep 09 20:40:39.058842 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.058817    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:40:39.058976 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.058910    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:40:39.068673 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.068641    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"]
Sep 09 20:40:39.152385 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.152152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c\" (UID: \"c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"
Sep 09 20:40:39.253164 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.253129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c\" (UID: \"c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"
Sep 09 20:40:39.253760 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.253739    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c\" (UID: \"c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"
Sep 09 20:40:39.372983 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.372895    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"
Sep 09 20:40:39.652293 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.652270    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"]
Sep 09 20:40:39.654453 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:40:39.654421    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77d7066_cdc3_4ffb_a3d7_bcd46fcf1f68.slice/crio-fcefb884295a21229fd0488d22e559546e93c9f4ec1f4f8d7aef5df61ab52140 WatchSource:0}: Error finding container fcefb884295a21229fd0488d22e559546e93c9f4ec1f4f8d7aef5df61ab52140: Status 404 returned error can't find the container with id fcefb884295a21229fd0488d22e559546e93c9f4ec1f4f8d7aef5df61ab52140
Sep 09 20:40:39.944410 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.944373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c" event={"ID":"c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68","Type":"ContainerStarted","Data":"5112a7faf2d204fb290c9adc3e808801ec2f2284ba2fa5e2d57dc35567033374"}
Sep 09 20:40:39.944591 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.944428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c" event={"ID":"c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68","Type":"ContainerStarted","Data":"fcefb884295a21229fd0488d22e559546e93c9f4ec1f4f8d7aef5df61ab52140"}
Sep 09 20:40:39.965066 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:39.965020    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c" podStartSLOduration=0.965001429 podStartE2EDuration="965.001429ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:39.963823313 +0000 UTC m=+1461.457034296" watchObservedRunningTime="2026-09-09 20:40:39.965001429 +0000 UTC m=+1461.458212413"
Sep 09 20:40:40.952025 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:40.951989    2581 generic.go:356] "Generic (PLEG): container finished" podID="c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68" containerID="5112a7faf2d204fb290c9adc3e808801ec2f2284ba2fa5e2d57dc35567033374" exitCode=0
Sep 09 20:40:40.952455 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:40.952080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c" event={"ID":"c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68","Type":"ContainerDied","Data":"5112a7faf2d204fb290c9adc3e808801ec2f2284ba2fa5e2d57dc35567033374"}
Sep 09 20:40:42.094939 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:42.094915    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"
Sep 09 20:40:42.179487 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:42.179449    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68-maas-api-service-ca\") pod \"c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68\" (UID: \"c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68\") "
Sep 09 20:40:42.180040 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:42.179867    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68" (UID: "c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:42.280334 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:42.280247    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68-maas-api-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:40:42.966426 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:42.966390    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c" event={"ID":"c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68","Type":"ContainerDied","Data":"fcefb884295a21229fd0488d22e559546e93c9f4ec1f4f8d7aef5df61ab52140"}
Sep 09 20:40:42.966605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:42.966464    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcefb884295a21229fd0488d22e559546e93c9f4ec1f4f8d7aef5df61ab52140"
Sep 09 20:40:42.966605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:42.966405    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"
Sep 09 20:40:54.585071 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.585033    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"]
Sep 09 20:40:54.585698 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.585288    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr" podUID="9a4818fc-961e-4572-a7aa-c2ca81f41c44" containerName="payload-processing" containerID="cri-o://48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c" gracePeriod=30
Sep 09 20:40:54.592249 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.592217    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"]
Sep 09 20:40:54.592539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.592512    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7" podUID="c07bb019-6d83-4a99-a32e-8c1a7c4ea57a" containerName="payload-pre-processing" containerID="cri-o://062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a" gracePeriod=30
Sep 09 20:40:54.842834 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.842748    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"]
Sep 09 20:40:54.846885 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.846851    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-06f7dc-4kb4c"]
Sep 09 20:40:54.875089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.875064    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:40:54.878493 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.878469    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:40:54.887082 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.887056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-plugins-config-volume\") pod \"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a\" (UID: \"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a\") "
Sep 09 20:40:54.887193 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.887087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6dtj\" (UniqueName: \"kubernetes.io/projected/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-kube-api-access-k6dtj\") pod \"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a\" (UID: \"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a\") "
Sep 09 20:40:54.887193 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.887106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tkbg\" (UniqueName: \"kubernetes.io/projected/9a4818fc-961e-4572-a7aa-c2ca81f41c44-kube-api-access-8tkbg\") pod \"9a4818fc-961e-4572-a7aa-c2ca81f41c44\" (UID: \"9a4818fc-961e-4572-a7aa-c2ca81f41c44\") "
Sep 09 20:40:54.887193 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.887150    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4818fc-961e-4572-a7aa-c2ca81f41c44-plugins-config-volume\") pod \"9a4818fc-961e-4572-a7aa-c2ca81f41c44\" (UID: \"9a4818fc-961e-4572-a7aa-c2ca81f41c44\") "
Sep 09 20:40:54.887604 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.887566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c07bb019-6d83-4a99-a32e-8c1a7c4ea57a" (UID: "c07bb019-6d83-4a99-a32e-8c1a7c4ea57a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:54.887696 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.887603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4818fc-961e-4572-a7aa-c2ca81f41c44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a4818fc-961e-4572-a7aa-c2ca81f41c44" (UID: "9a4818fc-961e-4572-a7aa-c2ca81f41c44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:40:54.889434 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.889406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-kube-api-access-k6dtj" (OuterVolumeSpecName: "kube-api-access-k6dtj") pod "c07bb019-6d83-4a99-a32e-8c1a7c4ea57a" (UID: "c07bb019-6d83-4a99-a32e-8c1a7c4ea57a"). InnerVolumeSpecName "kube-api-access-k6dtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:54.889491 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.889469    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4818fc-961e-4572-a7aa-c2ca81f41c44-kube-api-access-8tkbg" (OuterVolumeSpecName: "kube-api-access-8tkbg") pod "9a4818fc-961e-4572-a7aa-c2ca81f41c44" (UID: "9a4818fc-961e-4572-a7aa-c2ca81f41c44"). InnerVolumeSpecName "kube-api-access-8tkbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:54.987900 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.987858    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:40:54.987900 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.987892    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6dtj\" (UniqueName: \"kubernetes.io/projected/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a-kube-api-access-k6dtj\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:40:54.987900 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.987904    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tkbg\" (UniqueName: \"kubernetes.io/projected/9a4818fc-961e-4572-a7aa-c2ca81f41c44-kube-api-access-8tkbg\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:40:54.988148 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:54.987917    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4818fc-961e-4572-a7aa-c2ca81f41c44-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:40:55.009612 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.004011    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68" path="/var/lib/kubelet/pods/c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68/volumes"
Sep 09 20:40:55.038667 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.038633    2581 generic.go:356] "Generic (PLEG): container finished" podID="c07bb019-6d83-4a99-a32e-8c1a7c4ea57a" containerID="062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a" exitCode=0
Sep 09 20:40:55.038853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.038692    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"
Sep 09 20:40:55.038853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.038724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7" event={"ID":"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a","Type":"ContainerDied","Data":"062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a"}
Sep 09 20:40:55.038853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.038784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7" event={"ID":"c07bb019-6d83-4a99-a32e-8c1a7c4ea57a","Type":"ContainerDied","Data":"a54a33599005fc96c32c7599e72ca7e01bc0be02b78abb94d81290bb5276366d"}
Sep 09 20:40:55.038853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.038814    2581 scope.go:117] "RemoveContainer" containerID="062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a"
Sep 09 20:40:55.040201 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.040171    2581 generic.go:356] "Generic (PLEG): container finished" podID="9a4818fc-961e-4572-a7aa-c2ca81f41c44" containerID="48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c" exitCode=0
Sep 09 20:40:55.040317 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.040214    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"
Sep 09 20:40:55.040317 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.040266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr" event={"ID":"9a4818fc-961e-4572-a7aa-c2ca81f41c44","Type":"ContainerDied","Data":"48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c"}
Sep 09 20:40:55.040317 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.040310    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr" event={"ID":"9a4818fc-961e-4572-a7aa-c2ca81f41c44","Type":"ContainerDied","Data":"60ff49204cfc869182274d613abb83e32524f20b401e81cefe0ebf94dc82cf38"}
Sep 09 20:40:55.048363 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.048326    2581 scope.go:117] "RemoveContainer" containerID="062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a"
Sep 09 20:40:55.048702 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:40:55.048679    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a\": container with ID starting with 062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a not found: ID does not exist" containerID="062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a"
Sep 09 20:40:55.048746 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.048712    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a"} err="failed to get container status \"062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a\": rpc error: code = NotFound desc = could not find container \"062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a\": container with ID starting with 062ddd2ba797031bc94d3439654bfac53ac1cb413a6466993e6f39954849df1a not found: ID does not exist"
Sep 09 20:40:55.048746 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.048729    2581 scope.go:117] "RemoveContainer" containerID="48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c"
Sep 09 20:40:55.055806 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.055784    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"]
Sep 09 20:40:55.057938 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.057921    2581 scope.go:117] "RemoveContainer" containerID="48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c"
Sep 09 20:40:55.058232 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:40:55.058215    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c\": container with ID starting with 48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c not found: ID does not exist" containerID="48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c"
Sep 09 20:40:55.058291 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.058239    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c"} err="failed to get container status \"48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c\": rpc error: code = NotFound desc = could not find container \"48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c\": container with ID starting with 48548cb3af4b4fef79970da0695f489f149d56295b5f3a03d0f3ca98cd4fb46c not found: ID does not exist"
Sep 09 20:40:55.062154 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.062129    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-06f7dc-6f9698df4-6cnz7"]
Sep 09 20:40:55.071713 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.071667    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"]
Sep 09 20:40:55.075650 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:55.075627    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-06f7dc-6bc585d6b9-f9fhr"]
Sep 09 20:40:57.002808 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:57.002770    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a4818fc-961e-4572-a7aa-c2ca81f41c44" path="/var/lib/kubelet/pods/9a4818fc-961e-4572-a7aa-c2ca81f41c44/volumes"
Sep 09 20:40:57.003195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:40:57.003123    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c07bb019-6d83-4a99-a32e-8c1a7c4ea57a" path="/var/lib/kubelet/pods/c07bb019-6d83-4a99-a32e-8c1a7c4ea57a/volumes"
Sep 09 20:41:01.714614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.714578    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"]
Sep 09 20:41:01.715018 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.714957    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68" containerName="revoke-keys"
Sep 09 20:41:01.715018 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.714968    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68" containerName="revoke-keys"
Sep 09 20:41:01.715018 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.714981    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c07bb019-6d83-4a99-a32e-8c1a7c4ea57a" containerName="payload-pre-processing"
Sep 09 20:41:01.715018 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.714987    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c07bb019-6d83-4a99-a32e-8c1a7c4ea57a" containerName="payload-pre-processing"
Sep 09 20:41:01.715018 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.715002    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a4818fc-961e-4572-a7aa-c2ca81f41c44" containerName="payload-processing"
Sep 09 20:41:01.715018 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.715007    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a4818fc-961e-4572-a7aa-c2ca81f41c44" containerName="payload-processing"
Sep 09 20:41:01.715213 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.715064    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c77d7066-cdc3-4ffb-a3d7-bcd46fcf1f68" containerName="revoke-keys"
Sep 09 20:41:01.715213 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.715074    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c07bb019-6d83-4a99-a32e-8c1a7c4ea57a" containerName="payload-pre-processing"
Sep 09 20:41:01.715213 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.715082    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a4818fc-961e-4572-a7aa-c2ca81f41c44" containerName="payload-processing"
Sep 09 20:41:01.719493 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.719474    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"
Sep 09 20:41:01.722553 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.722526    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:41:01.722695 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.722589    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:41:01.724940 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.724904    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"]
Sep 09 20:41:01.750123 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.750089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6636e0-e874-44d6-9cd1-50c2f3884f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b\" (UID: \"3d6636e0-e874-44d6-9cd1-50c2f3884f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"
Sep 09 20:41:01.850862 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.850831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6636e0-e874-44d6-9cd1-50c2f3884f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b\" (UID: \"3d6636e0-e874-44d6-9cd1-50c2f3884f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"
Sep 09 20:41:01.851498 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:01.851477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6636e0-e874-44d6-9cd1-50c2f3884f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b\" (UID: \"3d6636e0-e874-44d6-9cd1-50c2f3884f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"
Sep 09 20:41:02.032446 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:02.032356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"
Sep 09 20:41:02.303178 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:02.303144    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"]
Sep 09 20:41:02.304604 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:41:02.304580    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6636e0_e874_44d6_9cd1_50c2f3884f9a.slice/crio-ea8c68cff1108f082f0d59bd227b9892ffa43a6dc636af4ca4b5f26855c54e13 WatchSource:0}: Error finding container ea8c68cff1108f082f0d59bd227b9892ffa43a6dc636af4ca4b5f26855c54e13: Status 404 returned error can't find the container with id ea8c68cff1108f082f0d59bd227b9892ffa43a6dc636af4ca4b5f26855c54e13
Sep 09 20:41:03.094487 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:03.094455    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d6636e0-e874-44d6-9cd1-50c2f3884f9a" containerID="6f9544fc64b15089ddc622a2a45e60d6306455184a00c34687ae32eba750e31c" exitCode=0
Sep 09 20:41:03.094833 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:03.094502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b" event={"ID":"3d6636e0-e874-44d6-9cd1-50c2f3884f9a","Type":"ContainerDied","Data":"6f9544fc64b15089ddc622a2a45e60d6306455184a00c34687ae32eba750e31c"}
Sep 09 20:41:03.094833 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:03.094552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b" event={"ID":"3d6636e0-e874-44d6-9cd1-50c2f3884f9a","Type":"ContainerStarted","Data":"ea8c68cff1108f082f0d59bd227b9892ffa43a6dc636af4ca4b5f26855c54e13"}
Sep 09 20:41:04.234686 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:04.234663    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"
Sep 09 20:41:04.273226 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:04.273186    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6636e0-e874-44d6-9cd1-50c2f3884f9a-maas-api-service-ca\") pod \"3d6636e0-e874-44d6-9cd1-50c2f3884f9a\" (UID: \"3d6636e0-e874-44d6-9cd1-50c2f3884f9a\") "
Sep 09 20:41:04.273555 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:04.273534    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6636e0-e874-44d6-9cd1-50c2f3884f9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d6636e0-e874-44d6-9cd1-50c2f3884f9a" (UID: "3d6636e0-e874-44d6-9cd1-50c2f3884f9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:04.374810 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:04.374721    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d6636e0-e874-44d6-9cd1-50c2f3884f9a-maas-api-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:05.108095 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:05.108064    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"
Sep 09 20:41:05.108265 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:05.108095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b" event={"ID":"3d6636e0-e874-44d6-9cd1-50c2f3884f9a","Type":"ContainerDied","Data":"ea8c68cff1108f082f0d59bd227b9892ffa43a6dc636af4ca4b5f26855c54e13"}
Sep 09 20:41:05.108265 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:05.108140    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea8c68cff1108f082f0d59bd227b9892ffa43a6dc636af4ca4b5f26855c54e13"
Sep 09 20:41:11.666190 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:11.666156    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"]
Sep 09 20:41:12.841724 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:12.841682    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"]
Sep 09 20:41:12.844229 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:12.844206    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc2ebb-lq24b"]
Sep 09 20:41:13.003191 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:13.003162    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d6636e0-e874-44d6-9cd1-50c2f3884f9a" path="/var/lib/kubelet/pods/3d6636e0-e874-44d6-9cd1-50c2f3884f9a/volumes"
Sep 09 20:41:19.070725 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.070685    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:41:19.073898 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.073875    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:41:19.280059 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.280027    2581 scope.go:117] "RemoveContainer" containerID="4addc4ec335d7e63f3aee94b77794ca0f81be92d5a429ae782b3b794a5489918"
Sep 09 20:41:19.288747 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.288726    2581 scope.go:117] "RemoveContainer" containerID="1116704d65418f90e030d185e1a60e07e2f903d079da5b610499a79977cc8028"
Sep 09 20:41:19.730251 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.730207    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"]
Sep 09 20:41:19.730684 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.730666    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d6636e0-e874-44d6-9cd1-50c2f3884f9a" containerName="revoke-keys"
Sep 09 20:41:19.730790 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.730687    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d6636e0-e874-44d6-9cd1-50c2f3884f9a" containerName="revoke-keys"
Sep 09 20:41:19.730842 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.730819    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d6636e0-e874-44d6-9cd1-50c2f3884f9a" containerName="revoke-keys"
Sep 09 20:41:19.735336 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.735315    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"
Sep 09 20:41:19.737784 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.737757    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:41:19.737902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.737763    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:41:19.742507 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.742485    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"]
Sep 09 20:41:19.823575 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.823536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7180977-8131-418a-bedf-e0b65eb17da7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6\" (UID: \"d7180977-8131-418a-bedf-e0b65eb17da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"
Sep 09 20:41:19.924953 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.924918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7180977-8131-418a-bedf-e0b65eb17da7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6\" (UID: \"d7180977-8131-418a-bedf-e0b65eb17da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"
Sep 09 20:41:19.925550 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:19.925517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7180977-8131-418a-bedf-e0b65eb17da7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6\" (UID: \"d7180977-8131-418a-bedf-e0b65eb17da7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"
Sep 09 20:41:20.048066 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:20.047980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"
Sep 09 20:41:20.320187 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:20.320164    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"]
Sep 09 20:41:20.322285 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:41:20.322260    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7180977_8131_418a_bedf_e0b65eb17da7.slice/crio-d0f4f462fe88be3f133535dfbac959a83dcd9def016ef00dfca17cbf2ff2a812 WatchSource:0}: Error finding container d0f4f462fe88be3f133535dfbac959a83dcd9def016ef00dfca17cbf2ff2a812: Status 404 returned error can't find the container with id d0f4f462fe88be3f133535dfbac959a83dcd9def016ef00dfca17cbf2ff2a812
Sep 09 20:41:21.203424 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:21.203391    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7180977-8131-418a-bedf-e0b65eb17da7" containerID="c9047e552dd088bccbf7c49fc2694e0048b3eb1f2ae37e1de37df9444dc47318" exitCode=0
Sep 09 20:41:21.203604 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:21.203434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6" event={"ID":"d7180977-8131-418a-bedf-e0b65eb17da7","Type":"ContainerDied","Data":"c9047e552dd088bccbf7c49fc2694e0048b3eb1f2ae37e1de37df9444dc47318"}
Sep 09 20:41:21.203604 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:21.203482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6" event={"ID":"d7180977-8131-418a-bedf-e0b65eb17da7","Type":"ContainerStarted","Data":"d0f4f462fe88be3f133535dfbac959a83dcd9def016ef00dfca17cbf2ff2a812"}
Sep 09 20:41:22.344734 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:22.344706    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"
Sep 09 20:41:22.452327 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:22.452286    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7180977-8131-418a-bedf-e0b65eb17da7-maas-api-service-ca\") pod \"d7180977-8131-418a-bedf-e0b65eb17da7\" (UID: \"d7180977-8131-418a-bedf-e0b65eb17da7\") "
Sep 09 20:41:22.452677 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:22.452653    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7180977-8131-418a-bedf-e0b65eb17da7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7180977-8131-418a-bedf-e0b65eb17da7" (UID: "d7180977-8131-418a-bedf-e0b65eb17da7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:22.553563 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:22.553479    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7180977-8131-418a-bedf-e0b65eb17da7-maas-api-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:23.217143 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:23.217103    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"
Sep 09 20:41:23.217290 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:23.217142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6" event={"ID":"d7180977-8131-418a-bedf-e0b65eb17da7","Type":"ContainerDied","Data":"d0f4f462fe88be3f133535dfbac959a83dcd9def016ef00dfca17cbf2ff2a812"}
Sep 09 20:41:23.217290 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:23.217192    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0f4f462fe88be3f133535dfbac959a83dcd9def016ef00dfca17cbf2ff2a812"
Sep 09 20:41:26.692519 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:26.692478    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r" podUID="a0a74beb-4f4d-4916-bc7d-3630ab9997ef" containerName="main" containerID="cri-o://845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a" gracePeriod=45
Sep 09 20:41:26.950788 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:26.950730    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:41:27.096146 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.096113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmkbn\" (UniqueName: \"kubernetes.io/projected/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-kube-api-access-zmkbn\") pod \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") "
Sep 09 20:41:27.096366 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.096166    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-home\") pod \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") "
Sep 09 20:41:27.096366 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.096199    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tmp-dir\") pod \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") "
Sep 09 20:41:27.096366 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.096226    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-dshm\") pod \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") "
Sep 09 20:41:27.096366 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.096251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-model-cache\") pod \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") "
Sep 09 20:41:27.096366 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.096294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tls-certs\") pod \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\" (UID: \"a0a74beb-4f4d-4916-bc7d-3630ab9997ef\") "
Sep 09 20:41:27.096610 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.096523    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-home" (OuterVolumeSpecName: "home") pod "a0a74beb-4f4d-4916-bc7d-3630ab9997ef" (UID: "a0a74beb-4f4d-4916-bc7d-3630ab9997ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:27.096610 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.096535    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0a74beb-4f4d-4916-bc7d-3630ab9997ef" (UID: "a0a74beb-4f4d-4916-bc7d-3630ab9997ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:27.096752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.096729    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tmp-dir\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:27.096805 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.096744    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0a74beb-4f4d-4916-bc7d-3630ab9997ef" (UID: "a0a74beb-4f4d-4916-bc7d-3630ab9997ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:27.096805 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.096758    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-home\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:27.098700 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.098677    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-dshm" (OuterVolumeSpecName: "dshm") pod "a0a74beb-4f4d-4916-bc7d-3630ab9997ef" (UID: "a0a74beb-4f4d-4916-bc7d-3630ab9997ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:41:27.098921 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.098903    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-kube-api-access-zmkbn" (OuterVolumeSpecName: "kube-api-access-zmkbn") pod "a0a74beb-4f4d-4916-bc7d-3630ab9997ef" (UID: "a0a74beb-4f4d-4916-bc7d-3630ab9997ef"). InnerVolumeSpecName "kube-api-access-zmkbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:27.098976 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.098913    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0a74beb-4f4d-4916-bc7d-3630ab9997ef" (UID: "a0a74beb-4f4d-4916-bc7d-3630ab9997ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:41:27.197549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.197513    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmkbn\" (UniqueName: \"kubernetes.io/projected/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-kube-api-access-zmkbn\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:27.197549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.197546    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-dshm\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:27.197549 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.197557    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-model-cache\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:27.197800 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.197565    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a74beb-4f4d-4916-bc7d-3630ab9997ef-tls-certs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:27.240990 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.240895    2581 generic.go:356] "Generic (PLEG): container finished" podID="a0a74beb-4f4d-4916-bc7d-3630ab9997ef" containerID="845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a" exitCode=0
Sep 09 20:41:27.240990 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.240936    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"
Sep 09 20:41:27.240990 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.240977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r" event={"ID":"a0a74beb-4f4d-4916-bc7d-3630ab9997ef","Type":"ContainerDied","Data":"845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a"}
Sep 09 20:41:27.241248 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.241023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r" event={"ID":"a0a74beb-4f4d-4916-bc7d-3630ab9997ef","Type":"ContainerDied","Data":"a0e0b817e991507deaeb69049e8b81cc262c391304dba3304441adcfd25285a7"}
Sep 09 20:41:27.241248 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.241041    2581 scope.go:117] "RemoveContainer" containerID="845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a"
Sep 09 20:41:27.251174 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.251154    2581 scope.go:117] "RemoveContainer" containerID="845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a"
Sep 09 20:41:27.251487 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:41:27.251468    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a\": container with ID starting with 845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a not found: ID does not exist" containerID="845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a"
Sep 09 20:41:27.251551 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.251497    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a"} err="failed to get container status \"845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a\": rpc error: code = NotFound desc = could not find container \"845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a\": container with ID starting with 845ee0bd41e0487c72b47636d8414bf23a6b2e3a96857e4d776755209b086c4a not found: ID does not exist"
Sep 09 20:41:27.262215 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.262188    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"]
Sep 09 20:41:27.269129 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:27.269083    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/sub-test-model-4101f2-kserve-859db84bc9-wsv8r"]
Sep 09 20:41:29.006843 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:29.006807    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a74beb-4f4d-4916-bc7d-3630ab9997ef" path="/var/lib/kubelet/pods/a0a74beb-4f4d-4916-bc7d-3630ab9997ef/volumes"
Sep 09 20:41:30.577285 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.577250    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"]
Sep 09 20:41:30.577772 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.577554    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k" podUID="2edc2a78-16ee-4f29-9969-2c70f26e969a" containerName="payload-processing" containerID="cri-o://a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c" gracePeriod=30
Sep 09 20:41:30.583462 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.583426    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"]
Sep 09 20:41:30.583703 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.583668    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw" podUID="aaf81c7f-b3eb-4631-9568-f841e18a7d9a" containerName="payload-pre-processing" containerID="cri-o://03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938" gracePeriod=30
Sep 09 20:41:30.758210 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.758175    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"]
Sep 09 20:41:30.762101 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.762069    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-47e11e-pqfs6"]
Sep 09 20:41:30.842671 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.842632    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:41:30.858859 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.858838    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:41:30.932312 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.932277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-plugins-config-volume\") pod \"aaf81c7f-b3eb-4631-9568-f841e18a7d9a\" (UID: \"aaf81c7f-b3eb-4631-9568-f841e18a7d9a\") "
Sep 09 20:41:30.932509 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.932356    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8wxt\" (UniqueName: \"kubernetes.io/projected/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-kube-api-access-z8wxt\") pod \"aaf81c7f-b3eb-4631-9568-f841e18a7d9a\" (UID: \"aaf81c7f-b3eb-4631-9568-f841e18a7d9a\") "
Sep 09 20:41:30.932705 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.932680    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aaf81c7f-b3eb-4631-9568-f841e18a7d9a" (UID: "aaf81c7f-b3eb-4631-9568-f841e18a7d9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:30.934558 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:30.934536    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-kube-api-access-z8wxt" (OuterVolumeSpecName: "kube-api-access-z8wxt") pod "aaf81c7f-b3eb-4631-9568-f841e18a7d9a" (UID: "aaf81c7f-b3eb-4631-9568-f841e18a7d9a"). InnerVolumeSpecName "kube-api-access-z8wxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:31.002650 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.002616    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7180977-8131-418a-bedf-e0b65eb17da7" path="/var/lib/kubelet/pods/d7180977-8131-418a-bedf-e0b65eb17da7/volumes"
Sep 09 20:41:31.033575 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.033545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2edc2a78-16ee-4f29-9969-2c70f26e969a-plugins-config-volume\") pod \"2edc2a78-16ee-4f29-9969-2c70f26e969a\" (UID: \"2edc2a78-16ee-4f29-9969-2c70f26e969a\") "
Sep 09 20:41:31.033706 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.033638    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvz47\" (UniqueName: \"kubernetes.io/projected/2edc2a78-16ee-4f29-9969-2c70f26e969a-kube-api-access-nvz47\") pod \"2edc2a78-16ee-4f29-9969-2c70f26e969a\" (UID: \"2edc2a78-16ee-4f29-9969-2c70f26e969a\") "
Sep 09 20:41:31.033827 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.033816    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:31.033865 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.033832    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8wxt\" (UniqueName: \"kubernetes.io/projected/aaf81c7f-b3eb-4631-9568-f841e18a7d9a-kube-api-access-z8wxt\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:31.033978 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.033953    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2edc2a78-16ee-4f29-9969-2c70f26e969a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2edc2a78-16ee-4f29-9969-2c70f26e969a" (UID: "2edc2a78-16ee-4f29-9969-2c70f26e969a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:41:31.035735 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.035713    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edc2a78-16ee-4f29-9969-2c70f26e969a-kube-api-access-nvz47" (OuterVolumeSpecName: "kube-api-access-nvz47") pod "2edc2a78-16ee-4f29-9969-2c70f26e969a" (UID: "2edc2a78-16ee-4f29-9969-2c70f26e969a"). InnerVolumeSpecName "kube-api-access-nvz47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:41:31.135257 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.135168    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2edc2a78-16ee-4f29-9969-2c70f26e969a-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:31.135257 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.135200    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvz47\" (UniqueName: \"kubernetes.io/projected/2edc2a78-16ee-4f29-9969-2c70f26e969a-kube-api-access-nvz47\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:41:31.267816 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.267776    2581 generic.go:356] "Generic (PLEG): container finished" podID="aaf81c7f-b3eb-4631-9568-f841e18a7d9a" containerID="03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938" exitCode=0
Sep 09 20:41:31.268020 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.267827    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"
Sep 09 20:41:31.268020 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.267868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw" event={"ID":"aaf81c7f-b3eb-4631-9568-f841e18a7d9a","Type":"ContainerDied","Data":"03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938"}
Sep 09 20:41:31.268020 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.267926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw" event={"ID":"aaf81c7f-b3eb-4631-9568-f841e18a7d9a","Type":"ContainerDied","Data":"7230d0f61389240b5fdd4e38dc48b320cbdf2ffa8cec78916320bdb40199af63"}
Sep 09 20:41:31.268020 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.267956    2581 scope.go:117] "RemoveContainer" containerID="03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938"
Sep 09 20:41:31.269512 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.269489    2581 generic.go:356] "Generic (PLEG): container finished" podID="2edc2a78-16ee-4f29-9969-2c70f26e969a" containerID="a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c" exitCode=0
Sep 09 20:41:31.269614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.269522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k" event={"ID":"2edc2a78-16ee-4f29-9969-2c70f26e969a","Type":"ContainerDied","Data":"a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c"}
Sep 09 20:41:31.269614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.269541    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"
Sep 09 20:41:31.269614 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.269578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k" event={"ID":"2edc2a78-16ee-4f29-9969-2c70f26e969a","Type":"ContainerDied","Data":"dd58f4a98e9a969a2ae9ee665efae1aa134c156eef24144ea74ee52245e5a7a2"}
Sep 09 20:41:31.277544 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.277527    2581 scope.go:117] "RemoveContainer" containerID="03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938"
Sep 09 20:41:31.277818 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:41:31.277798    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938\": container with ID starting with 03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938 not found: ID does not exist" containerID="03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938"
Sep 09 20:41:31.277903 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.277825    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938"} err="failed to get container status \"03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938\": rpc error: code = NotFound desc = could not find container \"03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938\": container with ID starting with 03b05b729278990a6d095ddf70abf0aecc1d429bf384d0a52993a3b5cd13d938 not found: ID does not exist"
Sep 09 20:41:31.277903 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.277840    2581 scope.go:117] "RemoveContainer" containerID="a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c"
Sep 09 20:41:31.286831 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.286804    2581 scope.go:117] "RemoveContainer" containerID="a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c"
Sep 09 20:41:31.287124 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:41:31.287105    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c\": container with ID starting with a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c not found: ID does not exist" containerID="a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c"
Sep 09 20:41:31.287174 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.287133    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c"} err="failed to get container status \"a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c\": rpc error: code = NotFound desc = could not find container \"a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c\": container with ID starting with a09f5fad8f9862d7a0f7ffc9842670356583b5c46687edf1b5307a4f4c96bd1c not found: ID does not exist"
Sep 09 20:41:31.287624 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.287605    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"]
Sep 09 20:41:31.297370 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.297313    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-47e11e-d7c64bff8-tg7mw"]
Sep 09 20:41:31.314042 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.314007    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"]
Sep 09 20:41:31.322476 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:31.322448    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-47e11e-58f867dc5c-p8x9k"]
Sep 09 20:41:33.002520 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:33.002488    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2edc2a78-16ee-4f29-9969-2c70f26e969a" path="/var/lib/kubelet/pods/2edc2a78-16ee-4f29-9969-2c70f26e969a/volumes"
Sep 09 20:41:33.002968 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:41:33.002831    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaf81c7f-b3eb-4631-9568-f841e18a7d9a" path="/var/lib/kubelet/pods/aaf81c7f-b3eb-4631-9568-f841e18a7d9a/volumes"
Sep 09 20:42:02.456270 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456238    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"]
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456597    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2edc2a78-16ee-4f29-9969-2c70f26e969a" containerName="payload-processing"
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456610    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2edc2a78-16ee-4f29-9969-2c70f26e969a" containerName="payload-processing"
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456617    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a74beb-4f4d-4916-bc7d-3630ab9997ef" containerName="main"
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456622    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a74beb-4f4d-4916-bc7d-3630ab9997ef" containerName="main"
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456646    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaf81c7f-b3eb-4631-9568-f841e18a7d9a" containerName="payload-pre-processing"
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456652    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaf81c7f-b3eb-4631-9568-f841e18a7d9a" containerName="payload-pre-processing"
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456658    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7180977-8131-418a-bedf-e0b65eb17da7" containerName="revoke-keys"
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456662    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7180977-8131-418a-bedf-e0b65eb17da7" containerName="revoke-keys"
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456714    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaf81c7f-b3eb-4631-9568-f841e18a7d9a" containerName="payload-pre-processing"
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456723    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a74beb-4f4d-4916-bc7d-3630ab9997ef" containerName="main"
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456729    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7180977-8131-418a-bedf-e0b65eb17da7" containerName="revoke-keys"
Sep 09 20:42:02.456761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.456736    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2edc2a78-16ee-4f29-9969-2c70f26e969a" containerName="payload-processing"
Sep 09 20:42:02.459236 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.459216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.462523 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.462498    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"rate-test-model-4101f2-kserve-self-signed-certs\""
Sep 09 20:42:02.463314 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.463295    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"kube-root-ca.crt\""
Sep 09 20:42:02.463381 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.463296    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"openshift-service-ca.crt\""
Sep 09 20:42:02.463433 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.463383    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4101f2\"/\"default-dockercfg-pmn8m\""
Sep 09 20:42:02.468977 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.468953    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"]
Sep 09 20:42:02.512475 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.512448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-home\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.512653 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.512490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-model-cache\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.512653 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.512558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tmp-dir\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.512653 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.512602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tls-certs\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.512653 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.512637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7db6p\" (UniqueName: \"kubernetes.io/projected/ae2b4534-7fd3-4c77-a646-4e44c9194da4-kube-api-access-7db6p\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.512812 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.512702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-dshm\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.613479 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.613447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tmp-dir\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.613657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.613494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tls-certs\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.613657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.613529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7db6p\" (UniqueName: \"kubernetes.io/projected/ae2b4534-7fd3-4c77-a646-4e44c9194da4-kube-api-access-7db6p\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.613657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.613563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-dshm\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.613657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.613588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-home\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.613657 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.613622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-model-cache\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.613917 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.613876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tmp-dir\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.614162 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.614140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-model-cache\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.614446 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.614179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-home\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.615828 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.615805    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-dshm\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.616194 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.616176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tls-certs\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.622433 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.622409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7db6p\" (UniqueName: \"kubernetes.io/projected/ae2b4534-7fd3-4c77-a646-4e44c9194da4-kube-api-access-7db6p\") pod \"rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") " pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:02.769452 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:02.769365    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:03.044655 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:42:03.044621    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae2b4534_7fd3_4c77_a646_4e44c9194da4.slice/crio-ac56276f9b4080eadf919c2b603a4eef8f78cf96c960439607ed0c8b49db0d43 WatchSource:0}: Error finding container ac56276f9b4080eadf919c2b603a4eef8f78cf96c960439607ed0c8b49db0d43: Status 404 returned error can't find the container with id ac56276f9b4080eadf919c2b603a4eef8f78cf96c960439607ed0c8b49db0d43
Sep 09 20:42:03.044888 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:03.044867    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"]
Sep 09 20:42:03.468163 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:03.468131    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j" event={"ID":"ae2b4534-7fd3-4c77-a646-4e44c9194da4","Type":"ContainerStarted","Data":"59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11"}
Sep 09 20:42:03.468574 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:03.468169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j" event={"ID":"ae2b4534-7fd3-4c77-a646-4e44c9194da4","Type":"ContainerStarted","Data":"ac56276f9b4080eadf919c2b603a4eef8f78cf96c960439607ed0c8b49db0d43"}
Sep 09 20:42:03.483972 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:03.483926    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j" podStartSLOduration=1.4839113720000001 podStartE2EDuration="1.483911372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:03.483873806 +0000 UTC m=+1544.977084789" watchObservedRunningTime="2026-09-09 20:42:03.483911372 +0000 UTC m=+1544.977122355"
Sep 09 20:42:12.769991 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:12.769950    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:12.769991 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:12.769999    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:12.781981 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:12.781956    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:13.537804 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:13.537775    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:19.330281 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:19.330250    2581 scope.go:117] "RemoveContainer" containerID="79ab5d00ce2e08a300bef5c45f79ff738d0c21ba6115790d533e01292f8a650d"
Sep 09 20:42:38.266481 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:38.266446    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"]
Sep 09 20:42:44.873170 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:44.873123    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"]
Sep 09 20:42:44.878285 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:44.878260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:42:44.880783 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:44.880763    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-868e40\""
Sep 09 20:42:44.885157 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:44.885135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"]
Sep 09 20:42:44.993921 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:44.993890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srtd\" (UniqueName: \"kubernetes.io/projected/fd7eab20-2a51-477d-bfe3-39e86975b107-kube-api-access-8srtd\") pod \"payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr\" (UID: \"fd7eab20-2a51-477d-bfe3-39e86975b107\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:42:44.994085 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:44.993931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7eab20-2a51-477d-bfe3-39e86975b107-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr\" (UID: \"fd7eab20-2a51-477d-bfe3-39e86975b107\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:42:45.095254 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:45.095210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srtd\" (UniqueName: \"kubernetes.io/projected/fd7eab20-2a51-477d-bfe3-39e86975b107-kube-api-access-8srtd\") pod \"payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr\" (UID: \"fd7eab20-2a51-477d-bfe3-39e86975b107\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:42:45.095254 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:45.095253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7eab20-2a51-477d-bfe3-39e86975b107-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr\" (UID: \"fd7eab20-2a51-477d-bfe3-39e86975b107\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:42:45.096101 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:45.096078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7eab20-2a51-477d-bfe3-39e86975b107-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr\" (UID: \"fd7eab20-2a51-477d-bfe3-39e86975b107\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:42:45.103335 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:45.103312    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srtd\" (UniqueName: \"kubernetes.io/projected/fd7eab20-2a51-477d-bfe3-39e86975b107-kube-api-access-8srtd\") pod \"payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr\" (UID: \"fd7eab20-2a51-477d-bfe3-39e86975b107\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:42:45.189884 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:45.189845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:42:45.461218 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:45.461185    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"]
Sep 09 20:42:45.462371 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:42:45.462328    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7eab20_2a51_477d_bfe3_39e86975b107.slice/crio-e96a3d84dab36b4d98200e69b685733ab7ddb879943ae6fc3178eae678b23c5c WatchSource:0}: Error finding container e96a3d84dab36b4d98200e69b685733ab7ddb879943ae6fc3178eae678b23c5c: Status 404 returned error can't find the container with id e96a3d84dab36b4d98200e69b685733ab7ddb879943ae6fc3178eae678b23c5c
Sep 09 20:42:45.716782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:45.716689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr" event={"ID":"fd7eab20-2a51-477d-bfe3-39e86975b107","Type":"ContainerStarted","Data":"77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c"}
Sep 09 20:42:45.716782 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:45.716740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr" event={"ID":"fd7eab20-2a51-477d-bfe3-39e86975b107","Type":"ContainerStarted","Data":"e96a3d84dab36b4d98200e69b685733ab7ddb879943ae6fc3178eae678b23c5c"}
Sep 09 20:42:45.717031 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:45.716814    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:42:45.734391 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:45.734321    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr" podStartSLOduration=1.734300545 podStartE2EDuration="1.734300545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:45.73160304 +0000 UTC m=+1587.224814053" watchObservedRunningTime="2026-09-09 20:42:45.734300545 +0000 UTC m=+1587.227511531"
Sep 09 20:42:53.287218 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.287150    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j" podUID="ae2b4534-7fd3-4c77-a646-4e44c9194da4" containerName="main" containerID="cri-o://59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11" gracePeriod=45
Sep 09 20:42:53.536745 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.536723    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:53.679192 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.679162    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tmp-dir\") pod \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") "
Sep 09 20:42:53.679395 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.679203    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-dshm\") pod \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") "
Sep 09 20:42:53.679395 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.679290    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7db6p\" (UniqueName: \"kubernetes.io/projected/ae2b4534-7fd3-4c77-a646-4e44c9194da4-kube-api-access-7db6p\") pod \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") "
Sep 09 20:42:53.679395 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.679310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tls-certs\") pod \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") "
Sep 09 20:42:53.679395 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.679330    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-home\") pod \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") "
Sep 09 20:42:53.679563 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.679520    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-model-cache\") pod \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\" (UID: \"ae2b4534-7fd3-4c77-a646-4e44c9194da4\") "
Sep 09 20:42:53.679621 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.679534    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae2b4534-7fd3-4c77-a646-4e44c9194da4" (UID: "ae2b4534-7fd3-4c77-a646-4e44c9194da4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:53.679621 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.679607    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-home" (OuterVolumeSpecName: "home") pod "ae2b4534-7fd3-4c77-a646-4e44c9194da4" (UID: "ae2b4534-7fd3-4c77-a646-4e44c9194da4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:53.679806 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.679776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae2b4534-7fd3-4c77-a646-4e44c9194da4" (UID: "ae2b4534-7fd3-4c77-a646-4e44c9194da4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:53.679888 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.679825    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-home\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:42:53.679888 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.679843    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tmp-dir\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:42:53.681842 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.681789    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae2b4534-7fd3-4c77-a646-4e44c9194da4" (UID: "ae2b4534-7fd3-4c77-a646-4e44c9194da4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:42:53.681934 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.681900    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-dshm" (OuterVolumeSpecName: "dshm") pod "ae2b4534-7fd3-4c77-a646-4e44c9194da4" (UID: "ae2b4534-7fd3-4c77-a646-4e44c9194da4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:53.681934 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.681915    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2b4534-7fd3-4c77-a646-4e44c9194da4-kube-api-access-7db6p" (OuterVolumeSpecName: "kube-api-access-7db6p") pod "ae2b4534-7fd3-4c77-a646-4e44c9194da4" (UID: "ae2b4534-7fd3-4c77-a646-4e44c9194da4"). InnerVolumeSpecName "kube-api-access-7db6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:42:53.767803 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.767769    2581 generic.go:356] "Generic (PLEG): container finished" podID="ae2b4534-7fd3-4c77-a646-4e44c9194da4" containerID="59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11" exitCode=0
Sep 09 20:42:53.767950 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.767813    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"
Sep 09 20:42:53.767950 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.767857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j" event={"ID":"ae2b4534-7fd3-4c77-a646-4e44c9194da4","Type":"ContainerDied","Data":"59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11"}
Sep 09 20:42:53.767950 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.767901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j" event={"ID":"ae2b4534-7fd3-4c77-a646-4e44c9194da4","Type":"ContainerDied","Data":"ac56276f9b4080eadf919c2b603a4eef8f78cf96c960439607ed0c8b49db0d43"}
Sep 09 20:42:53.767950 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.767918    2581 scope.go:117] "RemoveContainer" containerID="59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11"
Sep 09 20:42:53.777930 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.777913    2581 scope.go:117] "RemoveContainer" containerID="59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11"
Sep 09 20:42:53.778204 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:42:53.778186    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11\": container with ID starting with 59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11 not found: ID does not exist" containerID="59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11"
Sep 09 20:42:53.778240 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.778213    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11"} err="failed to get container status \"59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11\": rpc error: code = NotFound desc = could not find container \"59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11\": container with ID starting with 59b98d8d3ba5e700b23234d67533fad111212a8927f0f2841fa3a50af4da8c11 not found: ID does not exist"
Sep 09 20:42:53.781314 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.781230    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-dshm\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:42:53.781314 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.781249    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7db6p\" (UniqueName: \"kubernetes.io/projected/ae2b4534-7fd3-4c77-a646-4e44c9194da4-kube-api-access-7db6p\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:42:53.781314 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.781259    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2b4534-7fd3-4c77-a646-4e44c9194da4-tls-certs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:42:53.781314 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.781268    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2b4534-7fd3-4c77-a646-4e44c9194da4-model-cache\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:42:53.789944 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.789708    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"]
Sep 09 20:42:53.793508 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:53.793487    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j"]
Sep 09 20:42:55.002588 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:55.002554    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae2b4534-7fd3-4c77-a646-4e44c9194da4" path="/var/lib/kubelet/pods/ae2b4534-7fd3-4c77-a646-4e44c9194da4/volumes"
Sep 09 20:42:56.724228 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:56.724195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:42:58.528130 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:42:58.528090    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-4101f2/rate-test-model-4101f2-kserve-85f6dfdff9-5tk5j" podUID="ae2b4534-7fd3-4c77-a646-4e44c9194da4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:43:14.724757 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.724676    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"]
Sep 09 20:43:14.725214 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.725093    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2b4534-7fd3-4c77-a646-4e44c9194da4" containerName="main"
Sep 09 20:43:14.725214 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.725105    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2b4534-7fd3-4c77-a646-4e44c9194da4" containerName="main"
Sep 09 20:43:14.725214 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.725173    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae2b4534-7fd3-4c77-a646-4e44c9194da4" containerName="main"
Sep 09 20:43:14.729444 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.729427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:43:14.732303 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.732285    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-697f19\""
Sep 09 20:43:14.732393 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.732376    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-697f19-dockercfg-n6l4d\""
Sep 09 20:43:14.738664 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.738641    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"]
Sep 09 20:43:14.748695 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.748664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnx4\" (UniqueName: \"kubernetes.io/projected/c39a7452-c585-4949-852b-5655ef355270-kube-api-access-zlnx4\") pod \"payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g\" (UID: \"c39a7452-c585-4949-852b-5655ef355270\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:43:14.748816 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.748733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c39a7452-c585-4949-852b-5655ef355270-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g\" (UID: \"c39a7452-c585-4949-852b-5655ef355270\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:43:14.782527 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.782506    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"]
Sep 09 20:43:14.785952 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.785937    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:43:14.802889 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.802862    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"]
Sep 09 20:43:14.849230 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.849197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c39a7452-c585-4949-852b-5655ef355270-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g\" (UID: \"c39a7452-c585-4949-852b-5655ef355270\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:43:14.849405 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.849243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvlmk\" (UniqueName: \"kubernetes.io/projected/4ae999a1-2c38-459d-bf86-e4928c78a668-kube-api-access-qvlmk\") pod \"payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd\" (UID: \"4ae999a1-2c38-459d-bf86-e4928c78a668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:43:14.849405 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.849336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ae999a1-2c38-459d-bf86-e4928c78a668-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd\" (UID: \"4ae999a1-2c38-459d-bf86-e4928c78a668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:43:14.849481 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.849440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnx4\" (UniqueName: \"kubernetes.io/projected/c39a7452-c585-4949-852b-5655ef355270-kube-api-access-zlnx4\") pod \"payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g\" (UID: \"c39a7452-c585-4949-852b-5655ef355270\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:43:14.849920 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.849903    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c39a7452-c585-4949-852b-5655ef355270-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g\" (UID: \"c39a7452-c585-4949-852b-5655ef355270\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:43:14.869217 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.869192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnx4\" (UniqueName: \"kubernetes.io/projected/c39a7452-c585-4949-852b-5655ef355270-kube-api-access-zlnx4\") pod \"payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g\" (UID: \"c39a7452-c585-4949-852b-5655ef355270\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:43:14.950574 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.950527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvlmk\" (UniqueName: \"kubernetes.io/projected/4ae999a1-2c38-459d-bf86-e4928c78a668-kube-api-access-qvlmk\") pod \"payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd\" (UID: \"4ae999a1-2c38-459d-bf86-e4928c78a668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:43:14.950574 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.950586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ae999a1-2c38-459d-bf86-e4928c78a668-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd\" (UID: \"4ae999a1-2c38-459d-bf86-e4928c78a668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:43:14.951196 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.951177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ae999a1-2c38-459d-bf86-e4928c78a668-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd\" (UID: \"4ae999a1-2c38-459d-bf86-e4928c78a668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:43:14.959412 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:14.959384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvlmk\" (UniqueName: \"kubernetes.io/projected/4ae999a1-2c38-459d-bf86-e4928c78a668-kube-api-access-qvlmk\") pod \"payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd\" (UID: \"4ae999a1-2c38-459d-bf86-e4928c78a668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:43:15.041027 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.040955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:43:15.095850 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.095819    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:43:15.313112 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.312962    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"]
Sep 09 20:43:15.315773 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:43:15.315749    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39a7452_c585_4949_852b_5655ef355270.slice/crio-df18aa97acf52b60974bce1f4d8460156aabb68fbfb4b2cc53367c0fa502dcc0 WatchSource:0}: Error finding container df18aa97acf52b60974bce1f4d8460156aabb68fbfb4b2cc53367c0fa502dcc0: Status 404 returned error can't find the container with id df18aa97acf52b60974bce1f4d8460156aabb68fbfb4b2cc53367c0fa502dcc0
Sep 09 20:43:15.377293 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.377271    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"]
Sep 09 20:43:15.378324 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:43:15.378300    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae999a1_2c38_459d_bf86_e4928c78a668.slice/crio-4bc54d5defbd03a925d1c80970300837cd872ad7ae36f593cd91d580448215d4 WatchSource:0}: Error finding container 4bc54d5defbd03a925d1c80970300837cd872ad7ae36f593cd91d580448215d4: Status 404 returned error can't find the container with id 4bc54d5defbd03a925d1c80970300837cd872ad7ae36f593cd91d580448215d4
Sep 09 20:43:15.895798 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.895759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g" event={"ID":"c39a7452-c585-4949-852b-5655ef355270","Type":"ContainerStarted","Data":"bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50"}
Sep 09 20:43:15.896238 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.895932    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:43:15.896238 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.895964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g" event={"ID":"c39a7452-c585-4949-852b-5655ef355270","Type":"ContainerStarted","Data":"df18aa97acf52b60974bce1f4d8460156aabb68fbfb4b2cc53367c0fa502dcc0"}
Sep 09 20:43:15.897518 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.897495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd" event={"ID":"4ae999a1-2c38-459d-bf86-e4928c78a668","Type":"ContainerStarted","Data":"e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd"}
Sep 09 20:43:15.897637 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.897529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd" event={"ID":"4ae999a1-2c38-459d-bf86-e4928c78a668","Type":"ContainerStarted","Data":"4bc54d5defbd03a925d1c80970300837cd872ad7ae36f593cd91d580448215d4"}
Sep 09 20:43:15.897637 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.897551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:43:15.913845 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.913804    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g" podStartSLOduration=1.913790615 podStartE2EDuration="1.913790615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:43:15.912655324 +0000 UTC m=+1617.405866307" watchObservedRunningTime="2026-09-09 20:43:15.913790615 +0000 UTC m=+1617.407001596"
Sep 09 20:43:15.929827 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:15.929785    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd" podStartSLOduration=1.929772883 podStartE2EDuration="1.929772883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:43:15.927994858 +0000 UTC m=+1617.421205843" watchObservedRunningTime="2026-09-09 20:43:15.929772883 +0000 UTC m=+1617.422983866"
Sep 09 20:43:19.364833 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:19.364800    2581 scope.go:117] "RemoveContainer" containerID="27392997fdac519d5e6842f80824fe4cb25061495f5f14f0499f54530032c924"
Sep 09 20:43:26.905769 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:26.905736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:43:26.906140 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:26.905813    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:43:41.675492 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:41.675460    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zdk65_63bec830-c2e4-4a4a-a998-280cbae1baf7/payload-processing/0.log"
Sep 09 20:43:41.794329 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:41.794301    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd_4ae999a1-2c38-459d-bf86-e4928c78a668/payload-processing/0.log"
Sep 09 20:43:42.946493 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:42.946461    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"]
Sep 09 20:43:42.951283 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:42.951265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:42.954770 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:42.954747    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-868e40\"/\"kube-root-ca.crt\""
Sep 09 20:43:42.954938 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:42.954756    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-868e40\"/\"ipp-route-868e40-kserve-self-signed-certs\""
Sep 09 20:43:42.954938 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:42.954756    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-868e40\"/\"openshift-service-ca.crt\""
Sep 09 20:43:42.954938 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:42.954846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-868e40\"/\"default-dockercfg-jbzjv\""
Sep 09 20:43:42.960013 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:42.959989    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"]
Sep 09 20:43:43.098104 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.098067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-model-cache\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.098104 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.098110    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-home\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.098385 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.098185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tls-certs\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.098385 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.098242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482ch\" (UniqueName: \"kubernetes.io/projected/cf5592a6-debc-46c5-b7dd-88d37de23ad7-kube-api-access-482ch\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.098385 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.098334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tmp-dir\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.098385 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.098384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-dshm\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.199682 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.199595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-model-cache\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.199682 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.199637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-home\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.199682 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.199664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tls-certs\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.199682 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.199684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482ch\" (UniqueName: \"kubernetes.io/projected/cf5592a6-debc-46c5-b7dd-88d37de23ad7-kube-api-access-482ch\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.200108 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.199717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tmp-dir\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.200108 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.199737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-dshm\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.200108 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.200082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-model-cache\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.200231 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.200113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-home\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.200231 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.200179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tmp-dir\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.202109 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.202087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-dshm\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.202326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.202305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tls-certs\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.217185 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.217167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-482ch\" (UniqueName: \"kubernetes.io/projected/cf5592a6-debc-46c5-b7dd-88d37de23ad7-kube-api-access-482ch\") pod \"ipp-route-868e40-kserve-65cb77bf75-582kx\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") " pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.261248 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.261226    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:43.531876 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:43.531852    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"]
Sep 09 20:43:43.532970 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:43:43.532943    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf5592a6_debc_46c5_b7dd_88d37de23ad7.slice/crio-5f4658da346b3292fedf0bc4eacf0d4cdfa9fdd408ae9673f755f4c3f82567ed WatchSource:0}: Error finding container 5f4658da346b3292fedf0bc4eacf0d4cdfa9fdd408ae9673f755f4c3f82567ed: Status 404 returned error can't find the container with id 5f4658da346b3292fedf0bc4eacf0d4cdfa9fdd408ae9673f755f4c3f82567ed
Sep 09 20:43:44.064538 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:44.064497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx" event={"ID":"cf5592a6-debc-46c5-b7dd-88d37de23ad7","Type":"ContainerStarted","Data":"03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464"}
Sep 09 20:43:44.064922 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:44.064543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx" event={"ID":"cf5592a6-debc-46c5-b7dd-88d37de23ad7","Type":"ContainerStarted","Data":"5f4658da346b3292fedf0bc4eacf0d4cdfa9fdd408ae9673f755f4c3f82567ed"}
Sep 09 20:43:44.081601 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:44.081558    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx" podStartSLOduration=2.081542955 podStartE2EDuration="2.081542955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:43:44.080454915 +0000 UTC m=+1645.573665898" watchObservedRunningTime="2026-09-09 20:43:44.081542955 +0000 UTC m=+1645.574753938"
Sep 09 20:43:53.261929 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:53.261899    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:53.262433 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:53.261944    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:53.274844 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:53.274807    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:43:54.142486 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:43:54.142456    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:44:07.201804 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:07.201768    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zdk65_63bec830-c2e4-4a4a-a998-280cbae1baf7/payload-processing/0.log"
Sep 09 20:44:07.323038 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:07.322984    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd_4ae999a1-2c38-459d-bf86-e4928c78a668/payload-processing/0.log"
Sep 09 20:44:14.750386 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.750333    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"]
Sep 09 20:44:14.754077 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.754055    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:14.756857 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.756839    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-dd5d0a\""
Sep 09 20:44:14.756974 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.756877    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-dd5d0a-dockercfg-fdmnn\""
Sep 09 20:44:14.764580 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.764556    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"]
Sep 09 20:44:14.776918 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.776891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sd58\" (UniqueName: \"kubernetes.io/projected/535215f8-fff9-40ad-ab49-b7ab72741cdc-kube-api-access-2sd58\") pod \"payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74\" (UID: \"535215f8-fff9-40ad-ab49-b7ab72741cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:14.777038 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.776939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535215f8-fff9-40ad-ab49-b7ab72741cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74\" (UID: \"535215f8-fff9-40ad-ab49-b7ab72741cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:14.779120 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.779096    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"]
Sep 09 20:44:14.782457 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.782442    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:14.790607 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.790587    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"]
Sep 09 20:44:14.877965 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.877935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sd58\" (UniqueName: \"kubernetes.io/projected/535215f8-fff9-40ad-ab49-b7ab72741cdc-kube-api-access-2sd58\") pod \"payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74\" (UID: \"535215f8-fff9-40ad-ab49-b7ab72741cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:14.878156 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.877990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535215f8-fff9-40ad-ab49-b7ab72741cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74\" (UID: \"535215f8-fff9-40ad-ab49-b7ab72741cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:14.878156 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.878053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkmbn\" (UniqueName: \"kubernetes.io/projected/2a29948a-7ba9-43e3-a765-299201dd0a06-kube-api-access-qkmbn\") pod \"payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz\" (UID: \"2a29948a-7ba9-43e3-a765-299201dd0a06\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:14.878281 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.878184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a29948a-7ba9-43e3-a765-299201dd0a06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz\" (UID: \"2a29948a-7ba9-43e3-a765-299201dd0a06\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:14.878772 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.878749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535215f8-fff9-40ad-ab49-b7ab72741cdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74\" (UID: \"535215f8-fff9-40ad-ab49-b7ab72741cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:14.889184 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.889161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sd58\" (UniqueName: \"kubernetes.io/projected/535215f8-fff9-40ad-ab49-b7ab72741cdc-kube-api-access-2sd58\") pod \"payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74\" (UID: \"535215f8-fff9-40ad-ab49-b7ab72741cdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:14.979529 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.979490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkmbn\" (UniqueName: \"kubernetes.io/projected/2a29948a-7ba9-43e3-a765-299201dd0a06-kube-api-access-qkmbn\") pod \"payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz\" (UID: \"2a29948a-7ba9-43e3-a765-299201dd0a06\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:14.979737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.979543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a29948a-7ba9-43e3-a765-299201dd0a06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz\" (UID: \"2a29948a-7ba9-43e3-a765-299201dd0a06\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:14.980219 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.980200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a29948a-7ba9-43e3-a765-299201dd0a06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz\" (UID: \"2a29948a-7ba9-43e3-a765-299201dd0a06\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:14.988483 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:14.988455    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkmbn\" (UniqueName: \"kubernetes.io/projected/2a29948a-7ba9-43e3-a765-299201dd0a06-kube-api-access-qkmbn\") pod \"payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz\" (UID: \"2a29948a-7ba9-43e3-a765-299201dd0a06\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:15.066389 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:15.066281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:15.093854 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:15.093788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:15.340165 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:15.340134    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"]
Sep 09 20:44:15.341421 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:44:15.341397    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535215f8_fff9_40ad_ab49_b7ab72741cdc.slice/crio-1d79bcddd32a90ee8bf2ad557480e55af362450356b45cf29367b2aa0a188554 WatchSource:0}: Error finding container 1d79bcddd32a90ee8bf2ad557480e55af362450356b45cf29367b2aa0a188554: Status 404 returned error can't find the container with id 1d79bcddd32a90ee8bf2ad557480e55af362450356b45cf29367b2aa0a188554
Sep 09 20:44:15.377141 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:15.377105    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"]
Sep 09 20:44:15.386549 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:44:15.386509    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a29948a_7ba9_43e3_a765_299201dd0a06.slice/crio-ba62d8fe7099d78aec09354910943a704ecce9b6d5601e5639ab61829c5a4dc2 WatchSource:0}: Error finding container ba62d8fe7099d78aec09354910943a704ecce9b6d5601e5639ab61829c5a4dc2: Status 404 returned error can't find the container with id ba62d8fe7099d78aec09354910943a704ecce9b6d5601e5639ab61829c5a4dc2
Sep 09 20:44:16.258410 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:16.258373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz" event={"ID":"2a29948a-7ba9-43e3-a765-299201dd0a06","Type":"ContainerStarted","Data":"489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee"}
Sep 09 20:44:16.258906 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:16.258439    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:16.258906 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:16.258462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz" event={"ID":"2a29948a-7ba9-43e3-a765-299201dd0a06","Type":"ContainerStarted","Data":"ba62d8fe7099d78aec09354910943a704ecce9b6d5601e5639ab61829c5a4dc2"}
Sep 09 20:44:16.260077 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:16.260048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74" event={"ID":"535215f8-fff9-40ad-ab49-b7ab72741cdc","Type":"ContainerStarted","Data":"07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142"}
Sep 09 20:44:16.260193 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:16.260094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74" event={"ID":"535215f8-fff9-40ad-ab49-b7ab72741cdc","Type":"ContainerStarted","Data":"1d79bcddd32a90ee8bf2ad557480e55af362450356b45cf29367b2aa0a188554"}
Sep 09 20:44:16.260193 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:16.260134    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:16.273444 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:16.273395    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz" podStartSLOduration=2.273377389 podStartE2EDuration="2.273377389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:16.272457867 +0000 UTC m=+1677.765668851" watchObservedRunningTime="2026-09-09 20:44:16.273377389 +0000 UTC m=+1677.766588369"
Sep 09 20:44:16.290256 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:16.290207    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74" podStartSLOduration=2.290193498 podStartE2EDuration="2.290193498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:16.288819601 +0000 UTC m=+1677.782030587" watchObservedRunningTime="2026-09-09 20:44:16.290193498 +0000 UTC m=+1677.783404480"
Sep 09 20:44:27.268283 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:27.268246    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:27.268792 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:27.268313    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:52.669637 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:52.669602    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"]
Sep 09 20:44:52.670120 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:52.669838    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz" podUID="2a29948a-7ba9-43e3-a765-299201dd0a06" containerName="payload-processing" containerID="cri-o://489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee" gracePeriod=30
Sep 09 20:44:52.689244 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:52.689210    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"]
Sep 09 20:44:52.689507 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:52.689477    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74" podUID="535215f8-fff9-40ad-ab49-b7ab72741cdc" containerName="payload-pre-processing" containerID="cri-o://07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142" gracePeriod=30
Sep 09 20:44:52.954175 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:52.954152    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:52.975650 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:52.975620    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:53.036230 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.036197    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sd58\" (UniqueName: \"kubernetes.io/projected/535215f8-fff9-40ad-ab49-b7ab72741cdc-kube-api-access-2sd58\") pod \"535215f8-fff9-40ad-ab49-b7ab72741cdc\" (UID: \"535215f8-fff9-40ad-ab49-b7ab72741cdc\") "
Sep 09 20:44:53.036436 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.036242    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535215f8-fff9-40ad-ab49-b7ab72741cdc-plugins-config-volume\") pod \"535215f8-fff9-40ad-ab49-b7ab72741cdc\" (UID: \"535215f8-fff9-40ad-ab49-b7ab72741cdc\") "
Sep 09 20:44:53.036436 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.036267    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a29948a-7ba9-43e3-a765-299201dd0a06-plugins-config-volume\") pod \"2a29948a-7ba9-43e3-a765-299201dd0a06\" (UID: \"2a29948a-7ba9-43e3-a765-299201dd0a06\") "
Sep 09 20:44:53.036436 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.036330    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkmbn\" (UniqueName: \"kubernetes.io/projected/2a29948a-7ba9-43e3-a765-299201dd0a06-kube-api-access-qkmbn\") pod \"2a29948a-7ba9-43e3-a765-299201dd0a06\" (UID: \"2a29948a-7ba9-43e3-a765-299201dd0a06\") "
Sep 09 20:44:53.036704 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.036673    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535215f8-fff9-40ad-ab49-b7ab72741cdc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "535215f8-fff9-40ad-ab49-b7ab72741cdc" (UID: "535215f8-fff9-40ad-ab49-b7ab72741cdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:44:53.036842 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.036678    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a29948a-7ba9-43e3-a765-299201dd0a06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a29948a-7ba9-43e3-a765-299201dd0a06" (UID: "2a29948a-7ba9-43e3-a765-299201dd0a06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:44:53.038673 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.038646    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a29948a-7ba9-43e3-a765-299201dd0a06-kube-api-access-qkmbn" (OuterVolumeSpecName: "kube-api-access-qkmbn") pod "2a29948a-7ba9-43e3-a765-299201dd0a06" (UID: "2a29948a-7ba9-43e3-a765-299201dd0a06"). InnerVolumeSpecName "kube-api-access-qkmbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:44:53.038773 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.038653    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535215f8-fff9-40ad-ab49-b7ab72741cdc-kube-api-access-2sd58" (OuterVolumeSpecName: "kube-api-access-2sd58") pod "535215f8-fff9-40ad-ab49-b7ab72741cdc" (UID: "535215f8-fff9-40ad-ab49-b7ab72741cdc"). InnerVolumeSpecName "kube-api-access-2sd58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:44:53.137152 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.137104    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sd58\" (UniqueName: \"kubernetes.io/projected/535215f8-fff9-40ad-ab49-b7ab72741cdc-kube-api-access-2sd58\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:44:53.137152 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.137149    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/535215f8-fff9-40ad-ab49-b7ab72741cdc-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:44:53.137152 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.137161    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a29948a-7ba9-43e3-a765-299201dd0a06-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:44:53.137443 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.137170    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkmbn\" (UniqueName: \"kubernetes.io/projected/2a29948a-7ba9-43e3-a765-299201dd0a06-kube-api-access-qkmbn\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:44:53.478803 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.478766    2581 generic.go:356] "Generic (PLEG): container finished" podID="2a29948a-7ba9-43e3-a765-299201dd0a06" containerID="489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee" exitCode=0
Sep 09 20:44:53.479010 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.478813    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"
Sep 09 20:44:53.479010 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.478854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz" event={"ID":"2a29948a-7ba9-43e3-a765-299201dd0a06","Type":"ContainerDied","Data":"489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee"}
Sep 09 20:44:53.479010 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.478917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz" event={"ID":"2a29948a-7ba9-43e3-a765-299201dd0a06","Type":"ContainerDied","Data":"ba62d8fe7099d78aec09354910943a704ecce9b6d5601e5639ab61829c5a4dc2"}
Sep 09 20:44:53.479010 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.478941    2581 scope.go:117] "RemoveContainer" containerID="489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee"
Sep 09 20:44:53.480195 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.480169    2581 generic.go:356] "Generic (PLEG): container finished" podID="535215f8-fff9-40ad-ab49-b7ab72741cdc" containerID="07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142" exitCode=0
Sep 09 20:44:53.480333 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.480211    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"
Sep 09 20:44:53.480333 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.480310    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74" event={"ID":"535215f8-fff9-40ad-ab49-b7ab72741cdc","Type":"ContainerDied","Data":"07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142"}
Sep 09 20:44:53.480487 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.480367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74" event={"ID":"535215f8-fff9-40ad-ab49-b7ab72741cdc","Type":"ContainerDied","Data":"1d79bcddd32a90ee8bf2ad557480e55af362450356b45cf29367b2aa0a188554"}
Sep 09 20:44:53.490153 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.490123    2581 scope.go:117] "RemoveContainer" containerID="489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee"
Sep 09 20:44:53.490490 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:44:53.490470    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee\": container with ID starting with 489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee not found: ID does not exist" containerID="489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee"
Sep 09 20:44:53.490562 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.490498    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee"} err="failed to get container status \"489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee\": rpc error: code = NotFound desc = could not find container \"489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee\": container with ID starting with 489424e059d0ddd3738ed263015053fe620d5fb0b057f27e3b2e3e995a53f1ee not found: ID does not exist"
Sep 09 20:44:53.490562 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.490514    2581 scope.go:117] "RemoveContainer" containerID="07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142"
Sep 09 20:44:53.499031 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.499013    2581 scope.go:117] "RemoveContainer" containerID="07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142"
Sep 09 20:44:53.499292 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:44:53.499272    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142\": container with ID starting with 07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142 not found: ID does not exist" containerID="07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142"
Sep 09 20:44:53.499375 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.499298    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142"} err="failed to get container status \"07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142\": rpc error: code = NotFound desc = could not find container \"07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142\": container with ID starting with 07440b1f80948fa4e75edfd598c413d9c81ab38b71cd464870bb0f5695b05142 not found: ID does not exist"
Sep 09 20:44:53.505271 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.505246    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"]
Sep 09 20:44:53.509879 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.509858    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-dd5d0a-66556f4cc9-f2llz"]
Sep 09 20:44:53.521617 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.521594    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"]
Sep 09 20:44:53.529897 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:53.529871    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-dd5d0a-dbf695ff4-c4b74"]
Sep 09 20:44:55.002946 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:55.002911    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a29948a-7ba9-43e3-a765-299201dd0a06" path="/var/lib/kubelet/pods/2a29948a-7ba9-43e3-a765-299201dd0a06/volumes"
Sep 09 20:44:55.003337 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:44:55.003263    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="535215f8-fff9-40ad-ab49-b7ab72741cdc" path="/var/lib/kubelet/pods/535215f8-fff9-40ad-ab49-b7ab72741cdc/volumes"
Sep 09 20:45:00.137242 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.137185    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm"]
Sep 09 20:45:00.137727 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.137630    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="535215f8-fff9-40ad-ab49-b7ab72741cdc" containerName="payload-pre-processing"
Sep 09 20:45:00.137727 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.137647    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="535215f8-fff9-40ad-ab49-b7ab72741cdc" containerName="payload-pre-processing"
Sep 09 20:45:00.137727 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.137663    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a29948a-7ba9-43e3-a765-299201dd0a06" containerName="payload-processing"
Sep 09 20:45:00.137727 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.137668    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a29948a-7ba9-43e3-a765-299201dd0a06" containerName="payload-processing"
Sep 09 20:45:00.137959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.137744    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a29948a-7ba9-43e3-a765-299201dd0a06" containerName="payload-processing"
Sep 09 20:45:00.137959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.137754    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="535215f8-fff9-40ad-ab49-b7ab72741cdc" containerName="payload-pre-processing"
Sep 09 20:45:00.142292 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.142269    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w"]
Sep 09 20:45:00.142511 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.142422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm"
Sep 09 20:45:00.145479 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.145456    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jd4vq\""
Sep 09 20:45:00.145930 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.145908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w"
Sep 09 20:45:00.155173 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.155141    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm"]
Sep 09 20:45:00.156055 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.156037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm"
Sep 09 20:45:00.158353 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.158314    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w"]
Sep 09 20:45:00.164831 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.164813    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w"
Sep 09 20:45:00.455973 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.455913    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm"]
Sep 09 20:45:00.458610 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:45:00.458579    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bc94f3_2ef2_4c49_b911_bfc05dcb747d.slice/crio-b7dc158b845760924a43dcec785c8cc59924e89a7cc726d5d010055740c02121 WatchSource:0}: Error finding container b7dc158b845760924a43dcec785c8cc59924e89a7cc726d5d010055740c02121: Status 404 returned error can't find the container with id b7dc158b845760924a43dcec785c8cc59924e89a7cc726d5d010055740c02121
Sep 09 20:45:00.459528 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.459498    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w"]
Sep 09 20:45:00.461087 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:45:00.461063    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfb7581_8ecc_40bb_9d38_d5085d419b00.slice/crio-4d1c2c073b4fde24a11cbddd9c0c536344bfa278c7c70d758a7d38c1fcb2a75f WatchSource:0}: Error finding container 4d1c2c073b4fde24a11cbddd9c0c536344bfa278c7c70d758a7d38c1fcb2a75f: Status 404 returned error can't find the container with id 4d1c2c073b4fde24a11cbddd9c0c536344bfa278c7c70d758a7d38c1fcb2a75f
Sep 09 20:45:00.524859 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.524828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w" event={"ID":"5cfb7581-8ecc-40bb-9d38-d5085d419b00","Type":"ContainerStarted","Data":"4d1c2c073b4fde24a11cbddd9c0c536344bfa278c7c70d758a7d38c1fcb2a75f"}
Sep 09 20:45:00.526054 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:00.526031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm" event={"ID":"99bc94f3-2ef2-4c49-b911-bfc05dcb747d","Type":"ContainerStarted","Data":"b7dc158b845760924a43dcec785c8cc59924e89a7cc726d5d010055740c02121"}
Sep 09 20:45:01.532849 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:01.532817    2581 generic.go:356] "Generic (PLEG): container finished" podID="5cfb7581-8ecc-40bb-9d38-d5085d419b00" containerID="642de41c8c3bc0e223dc2b053ad1569c57592bf984abf29cb96baf1556effb8e" exitCode=0
Sep 09 20:45:01.533282 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:01.532906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w" event={"ID":"5cfb7581-8ecc-40bb-9d38-d5085d419b00","Type":"ContainerDied","Data":"642de41c8c3bc0e223dc2b053ad1569c57592bf984abf29cb96baf1556effb8e"}
Sep 09 20:45:01.534565 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:01.534543    2581 generic.go:356] "Generic (PLEG): container finished" podID="99bc94f3-2ef2-4c49-b911-bfc05dcb747d" containerID="82dfb53bd3b69cd911e5e596887ebbe96f9b82205ef3f9ffd8df21978d8fbffa" exitCode=0
Sep 09 20:45:01.534684 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:01.534576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm" event={"ID":"99bc94f3-2ef2-4c49-b911-bfc05dcb747d","Type":"ContainerDied","Data":"82dfb53bd3b69cd911e5e596887ebbe96f9b82205ef3f9ffd8df21978d8fbffa"}
Sep 09 20:45:02.694605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.694581    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w"
Sep 09 20:45:02.697807 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.697752    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm"
Sep 09 20:45:02.918460 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.918376    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"]
Sep 09 20:45:02.918793 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.918781    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99bc94f3-2ef2-4c49-b911-bfc05dcb747d" containerName="cleanup"
Sep 09 20:45:02.918835 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.918796    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="99bc94f3-2ef2-4c49-b911-bfc05dcb747d" containerName="cleanup"
Sep 09 20:45:02.918835 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.918817    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cfb7581-8ecc-40bb-9d38-d5085d419b00" containerName="cleanup"
Sep 09 20:45:02.918835 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.918826    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cfb7581-8ecc-40bb-9d38-d5085d419b00" containerName="cleanup"
Sep 09 20:45:02.918924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.918891    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="99bc94f3-2ef2-4c49-b911-bfc05dcb747d" containerName="cleanup"
Sep 09 20:45:02.918924 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.918902    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cfb7581-8ecc-40bb-9d38-d5085d419b00" containerName="cleanup"
Sep 09 20:45:02.921950 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.921935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"
Sep 09 20:45:02.924593 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.924575    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:45:02.929603 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:02.929235    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"]
Sep 09 20:45:03.030363 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.030314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99502b67-71fc-4dfd-9f61-924adb4317fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg\" (UID: \"99502b67-71fc-4dfd-9f61-924adb4317fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"
Sep 09 20:45:03.131815 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.131777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99502b67-71fc-4dfd-9f61-924adb4317fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg\" (UID: \"99502b67-71fc-4dfd-9f61-924adb4317fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"
Sep 09 20:45:03.132440 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.132421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99502b67-71fc-4dfd-9f61-924adb4317fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg\" (UID: \"99502b67-71fc-4dfd-9f61-924adb4317fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"
Sep 09 20:45:03.233155 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.233117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"
Sep 09 20:45:03.511085 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.511058    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"]
Sep 09 20:45:03.513291 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:45:03.513257    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99502b67_71fc_4dfd_9f61_924adb4317fa.slice/crio-6a0acc7386a0c947033ca8a10fa0af4a6fd1c2ed6111b9d671b2a76d8627e183 WatchSource:0}: Error finding container 6a0acc7386a0c947033ca8a10fa0af4a6fd1c2ed6111b9d671b2a76d8627e183: Status 404 returned error can't find the container with id 6a0acc7386a0c947033ca8a10fa0af4a6fd1c2ed6111b9d671b2a76d8627e183
Sep 09 20:45:03.548160 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.548125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg" event={"ID":"99502b67-71fc-4dfd-9f61-924adb4317fa","Type":"ContainerStarted","Data":"6a0acc7386a0c947033ca8a10fa0af4a6fd1c2ed6111b9d671b2a76d8627e183"}
Sep 09 20:45:03.549335 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.549315    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w"
Sep 09 20:45:03.549457 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.549355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w" event={"ID":"5cfb7581-8ecc-40bb-9d38-d5085d419b00","Type":"ContainerDied","Data":"4d1c2c073b4fde24a11cbddd9c0c536344bfa278c7c70d758a7d38c1fcb2a75f"}
Sep 09 20:45:03.549457 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.549404    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d1c2c073b4fde24a11cbddd9c0c536344bfa278c7c70d758a7d38c1fcb2a75f"
Sep 09 20:45:03.550563 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.550540    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm"
Sep 09 20:45:03.550651 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.550591    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm" event={"ID":"99bc94f3-2ef2-4c49-b911-bfc05dcb747d","Type":"ContainerDied","Data":"b7dc158b845760924a43dcec785c8cc59924e89a7cc726d5d010055740c02121"}
Sep 09 20:45:03.550651 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:03.550625    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7dc158b845760924a43dcec785c8cc59924e89a7cc726d5d010055740c02121"
Sep 09 20:45:04.557975 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:04.557885    2581 generic.go:356] "Generic (PLEG): container finished" podID="99502b67-71fc-4dfd-9f61-924adb4317fa" containerID="328d49a91c32430e0d51042e8908e97cf5f4c565b2e006691e7f2093e1dfa194" exitCode=0
Sep 09 20:45:04.557975 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:04.557953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg" event={"ID":"99502b67-71fc-4dfd-9f61-924adb4317fa","Type":"ContainerDied","Data":"328d49a91c32430e0d51042e8908e97cf5f4c565b2e006691e7f2093e1dfa194"}
Sep 09 20:45:05.696947 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:05.696919    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"
Sep 09 20:45:05.756742 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:05.756714    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99502b67-71fc-4dfd-9f61-924adb4317fa-maas-api-service-ca\") pod \"99502b67-71fc-4dfd-9f61-924adb4317fa\" (UID: \"99502b67-71fc-4dfd-9f61-924adb4317fa\") "
Sep 09 20:45:05.757081 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:05.757057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99502b67-71fc-4dfd-9f61-924adb4317fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99502b67-71fc-4dfd-9f61-924adb4317fa" (UID: "99502b67-71fc-4dfd-9f61-924adb4317fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:45:05.858144 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:05.858057    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99502b67-71fc-4dfd-9f61-924adb4317fa-maas-api-service-ca\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:45:06.572089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:06.572051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg" event={"ID":"99502b67-71fc-4dfd-9f61-924adb4317fa","Type":"ContainerDied","Data":"6a0acc7386a0c947033ca8a10fa0af4a6fd1c2ed6111b9d671b2a76d8627e183"}
Sep 09 20:45:06.572089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:06.572085    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"
Sep 09 20:45:06.572297 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:06.572114    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a0acc7386a0c947033ca8a10fa0af4a6fd1c2ed6111b9d671b2a76d8627e183"
Sep 09 20:45:18.527349 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:18.527316    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"]
Sep 09 20:45:18.527804 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:18.527575    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr" podUID="fd7eab20-2a51-477d-bfe3-39e86975b107" containerName="payload-pre-processing" containerID="cri-o://77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c" gracePeriod=30
Sep 09 20:45:18.722240 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:18.722205    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"]
Sep 09 20:45:18.724618 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:18.724589    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-868e40-jpqhg"]
Sep 09 20:45:18.779802 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:18.779748    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:45:18.877415 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:18.877379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7eab20-2a51-477d-bfe3-39e86975b107-plugins-config-volume\") pod \"fd7eab20-2a51-477d-bfe3-39e86975b107\" (UID: \"fd7eab20-2a51-477d-bfe3-39e86975b107\") "
Sep 09 20:45:18.877562 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:18.877475    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8srtd\" (UniqueName: \"kubernetes.io/projected/fd7eab20-2a51-477d-bfe3-39e86975b107-kube-api-access-8srtd\") pod \"fd7eab20-2a51-477d-bfe3-39e86975b107\" (UID: \"fd7eab20-2a51-477d-bfe3-39e86975b107\") "
Sep 09 20:45:18.877779 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:18.877758    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7eab20-2a51-477d-bfe3-39e86975b107-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd7eab20-2a51-477d-bfe3-39e86975b107" (UID: "fd7eab20-2a51-477d-bfe3-39e86975b107"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:45:18.879575 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:18.879557    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7eab20-2a51-477d-bfe3-39e86975b107-kube-api-access-8srtd" (OuterVolumeSpecName: "kube-api-access-8srtd") pod "fd7eab20-2a51-477d-bfe3-39e86975b107" (UID: "fd7eab20-2a51-477d-bfe3-39e86975b107"). InnerVolumeSpecName "kube-api-access-8srtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:18.978705 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:18.978671    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8srtd\" (UniqueName: \"kubernetes.io/projected/fd7eab20-2a51-477d-bfe3-39e86975b107-kube-api-access-8srtd\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:45:18.978705 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:18.978698    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7eab20-2a51-477d-bfe3-39e86975b107-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:45:19.003082 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:19.003038    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99502b67-71fc-4dfd-9f61-924adb4317fa" path="/var/lib/kubelet/pods/99502b67-71fc-4dfd-9f61-924adb4317fa/volumes"
Sep 09 20:45:19.648513 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:19.648476    2581 generic.go:356] "Generic (PLEG): container finished" podID="fd7eab20-2a51-477d-bfe3-39e86975b107" containerID="77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c" exitCode=0
Sep 09 20:45:19.648513 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:19.648516    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"
Sep 09 20:45:19.648984 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:19.648567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr" event={"ID":"fd7eab20-2a51-477d-bfe3-39e86975b107","Type":"ContainerDied","Data":"77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c"}
Sep 09 20:45:19.648984 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:19.648615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr" event={"ID":"fd7eab20-2a51-477d-bfe3-39e86975b107","Type":"ContainerDied","Data":"e96a3d84dab36b4d98200e69b685733ab7ddb879943ae6fc3178eae678b23c5c"}
Sep 09 20:45:19.648984 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:19.648636    2581 scope.go:117] "RemoveContainer" containerID="77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c"
Sep 09 20:45:19.657596 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:19.657563    2581 scope.go:117] "RemoveContainer" containerID="77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c"
Sep 09 20:45:19.657814 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:45:19.657797    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c\": container with ID starting with 77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c not found: ID does not exist" containerID="77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c"
Sep 09 20:45:19.657854 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:19.657821    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c"} err="failed to get container status \"77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c\": rpc error: code = NotFound desc = could not find container \"77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c\": container with ID starting with 77543c731a7a97ec9cc16069fa8cd6e990f05f0fbc0acff15c04e9e0262c1a8c not found: ID does not exist"
Sep 09 20:45:19.672023 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:19.671999    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"]
Sep 09 20:45:19.675096 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:19.675076    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-868e40-5df55fbdc8-fq7pr"]
Sep 09 20:45:21.002629 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:21.002594    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd7eab20-2a51-477d-bfe3-39e86975b107" path="/var/lib/kubelet/pods/fd7eab20-2a51-477d-bfe3-39e86975b107/volumes"
Sep 09 20:45:24.055638 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:24.055604    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"]
Sep 09 20:45:39.075448 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.075405    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx" podUID="cf5592a6-debc-46c5-b7dd-88d37de23ad7" containerName="main" containerID="cri-o://03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464" gracePeriod=45
Sep 09 20:45:39.325326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.325298    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:45:39.359754 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.359664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-home\") pod \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") "
Sep 09 20:45:39.359754 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.359711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-model-cache\") pod \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") "
Sep 09 20:45:39.359959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.359757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tls-certs\") pod \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") "
Sep 09 20:45:39.359959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.359785    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-dshm\") pod \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") "
Sep 09 20:45:39.359959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.359820    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tmp-dir\") pod \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") "
Sep 09 20:45:39.359959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.359880    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-482ch\" (UniqueName: \"kubernetes.io/projected/cf5592a6-debc-46c5-b7dd-88d37de23ad7-kube-api-access-482ch\") pod \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\" (UID: \"cf5592a6-debc-46c5-b7dd-88d37de23ad7\") "
Sep 09 20:45:39.359959 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.359916    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-home" (OuterVolumeSpecName: "home") pod "cf5592a6-debc-46c5-b7dd-88d37de23ad7" (UID: "cf5592a6-debc-46c5-b7dd-88d37de23ad7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:39.360190 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.360052    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf5592a6-debc-46c5-b7dd-88d37de23ad7" (UID: "cf5592a6-debc-46c5-b7dd-88d37de23ad7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:39.360190 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.360125    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf5592a6-debc-46c5-b7dd-88d37de23ad7" (UID: "cf5592a6-debc-46c5-b7dd-88d37de23ad7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:39.360287 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.360204    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-home\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:45:39.360287 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.360224    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-model-cache\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:45:39.360287 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.360238    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tmp-dir\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:45:39.362373 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.362200    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf5592a6-debc-46c5-b7dd-88d37de23ad7" (UID: "cf5592a6-debc-46c5-b7dd-88d37de23ad7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:45:39.362373 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.362314    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5592a6-debc-46c5-b7dd-88d37de23ad7-kube-api-access-482ch" (OuterVolumeSpecName: "kube-api-access-482ch") pod "cf5592a6-debc-46c5-b7dd-88d37de23ad7" (UID: "cf5592a6-debc-46c5-b7dd-88d37de23ad7"). InnerVolumeSpecName "kube-api-access-482ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:45:39.362662 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.362633    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-dshm" (OuterVolumeSpecName: "dshm") pod "cf5592a6-debc-46c5-b7dd-88d37de23ad7" (UID: "cf5592a6-debc-46c5-b7dd-88d37de23ad7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:45:39.461617 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.461580    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-482ch\" (UniqueName: \"kubernetes.io/projected/cf5592a6-debc-46c5-b7dd-88d37de23ad7-kube-api-access-482ch\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:45:39.461617 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.461609    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5592a6-debc-46c5-b7dd-88d37de23ad7-tls-certs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:45:39.461617 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.461619    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5592a6-debc-46c5-b7dd-88d37de23ad7-dshm\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:45:39.769418 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.769378    2581 generic.go:356] "Generic (PLEG): container finished" podID="cf5592a6-debc-46c5-b7dd-88d37de23ad7" containerID="03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464" exitCode=0
Sep 09 20:45:39.769605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.769433    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"
Sep 09 20:45:39.769605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.769462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx" event={"ID":"cf5592a6-debc-46c5-b7dd-88d37de23ad7","Type":"ContainerDied","Data":"03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464"}
Sep 09 20:45:39.769605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.769508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx" event={"ID":"cf5592a6-debc-46c5-b7dd-88d37de23ad7","Type":"ContainerDied","Data":"5f4658da346b3292fedf0bc4eacf0d4cdfa9fdd408ae9673f755f4c3f82567ed"}
Sep 09 20:45:39.769605 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.769527    2581 scope.go:117] "RemoveContainer" containerID="03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464"
Sep 09 20:45:39.779238 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.779224    2581 scope.go:117] "RemoveContainer" containerID="03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464"
Sep 09 20:45:39.779534 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:45:39.779515    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464\": container with ID starting with 03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464 not found: ID does not exist" containerID="03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464"
Sep 09 20:45:39.779592 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.779542    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464"} err="failed to get container status \"03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464\": rpc error: code = NotFound desc = could not find container \"03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464\": container with ID starting with 03bd0ba2886bb8516155c012d112b8274eaf3a33ad32ccf08274f6c1d781f464 not found: ID does not exist"
Sep 09 20:45:39.792423 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.792402    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"]
Sep 09 20:45:39.796178 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:39.796150    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-868e40/ipp-route-868e40-kserve-65cb77bf75-582kx"]
Sep 09 20:45:41.002206 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:45:41.002171    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf5592a6-debc-46c5-b7dd-88d37de23ad7" path="/var/lib/kubelet/pods/cf5592a6-debc-46c5-b7dd-88d37de23ad7/volumes"
Sep 09 20:46:19.110799 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:19.110771    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:46:19.116121 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:19.116100    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:46:19.415869 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:19.415754    2581 scope.go:117] "RemoveContainer" containerID="8c6d4c67b0984a6577e739df1c52933cb87939e387ba6d100b7eed61b6604f16"
Sep 09 20:46:20.213388 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.213354    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm"]
Sep 09 20:46:20.217032 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.217003    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-697f19-29816445-6fvsm"]
Sep 09 20:46:20.280974 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.280943    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"]
Sep 09 20:46:20.281163 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.281129    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd" podUID="4ae999a1-2c38-459d-bf86-e4928c78a668" containerName="payload-processing" containerID="cri-o://e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd" gracePeriod=30
Sep 09 20:46:20.310094 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.310064    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"]
Sep 09 20:46:20.310434 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.310404    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g" podUID="c39a7452-c585-4949-852b-5655ef355270" containerName="payload-pre-processing" containerID="cri-o://bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50" gracePeriod=30
Sep 09 20:46:20.563853 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.563828    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:46:20.596369 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.596309    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:46:20.727169 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.727081    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ae999a1-2c38-459d-bf86-e4928c78a668-plugins-config-volume\") pod \"4ae999a1-2c38-459d-bf86-e4928c78a668\" (UID: \"4ae999a1-2c38-459d-bf86-e4928c78a668\") "
Sep 09 20:46:20.727169 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.727151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c39a7452-c585-4949-852b-5655ef355270-plugins-config-volume\") pod \"c39a7452-c585-4949-852b-5655ef355270\" (UID: \"c39a7452-c585-4949-852b-5655ef355270\") "
Sep 09 20:46:20.727443 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.727187    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvlmk\" (UniqueName: \"kubernetes.io/projected/4ae999a1-2c38-459d-bf86-e4928c78a668-kube-api-access-qvlmk\") pod \"4ae999a1-2c38-459d-bf86-e4928c78a668\" (UID: \"4ae999a1-2c38-459d-bf86-e4928c78a668\") "
Sep 09 20:46:20.727443 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.727249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlnx4\" (UniqueName: \"kubernetes.io/projected/c39a7452-c585-4949-852b-5655ef355270-kube-api-access-zlnx4\") pod \"c39a7452-c585-4949-852b-5655ef355270\" (UID: \"c39a7452-c585-4949-852b-5655ef355270\") "
Sep 09 20:46:20.727566 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.727525    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae999a1-2c38-459d-bf86-e4928c78a668-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ae999a1-2c38-459d-bf86-e4928c78a668" (UID: "4ae999a1-2c38-459d-bf86-e4928c78a668"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:20.727664 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.727640    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39a7452-c585-4949-852b-5655ef355270-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c39a7452-c585-4949-852b-5655ef355270" (UID: "c39a7452-c585-4949-852b-5655ef355270"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:20.729495 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.729472    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae999a1-2c38-459d-bf86-e4928c78a668-kube-api-access-qvlmk" (OuterVolumeSpecName: "kube-api-access-qvlmk") pod "4ae999a1-2c38-459d-bf86-e4928c78a668" (UID: "4ae999a1-2c38-459d-bf86-e4928c78a668"). InnerVolumeSpecName "kube-api-access-qvlmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:20.729783 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.729760    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39a7452-c585-4949-852b-5655ef355270-kube-api-access-zlnx4" (OuterVolumeSpecName: "kube-api-access-zlnx4") pod "c39a7452-c585-4949-852b-5655ef355270" (UID: "c39a7452-c585-4949-852b-5655ef355270"). InnerVolumeSpecName "kube-api-access-zlnx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:20.828717 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.828677    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvlmk\" (UniqueName: \"kubernetes.io/projected/4ae999a1-2c38-459d-bf86-e4928c78a668-kube-api-access-qvlmk\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:46:20.828717 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.828712    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlnx4\" (UniqueName: \"kubernetes.io/projected/c39a7452-c585-4949-852b-5655ef355270-kube-api-access-zlnx4\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:46:20.828717 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.828721    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ae999a1-2c38-459d-bf86-e4928c78a668-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:46:20.828717 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:20.828729    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c39a7452-c585-4949-852b-5655ef355270-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:46:21.003326 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.003236    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99bc94f3-2ef2-4c49-b911-bfc05dcb747d" path="/var/lib/kubelet/pods/99bc94f3-2ef2-4c49-b911-bfc05dcb747d/volumes"
Sep 09 20:46:21.016373 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.016324    2581 generic.go:356] "Generic (PLEG): container finished" podID="c39a7452-c585-4949-852b-5655ef355270" containerID="bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50" exitCode=0
Sep 09 20:46:21.016491 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.016370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g" event={"ID":"c39a7452-c585-4949-852b-5655ef355270","Type":"ContainerDied","Data":"bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50"}
Sep 09 20:46:21.016491 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.016387    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"
Sep 09 20:46:21.016491 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.016424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g" event={"ID":"c39a7452-c585-4949-852b-5655ef355270","Type":"ContainerDied","Data":"df18aa97acf52b60974bce1f4d8460156aabb68fbfb4b2cc53367c0fa502dcc0"}
Sep 09 20:46:21.016491 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.016446    2581 scope.go:117] "RemoveContainer" containerID="bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50"
Sep 09 20:46:21.017788 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.017763    2581 generic.go:356] "Generic (PLEG): container finished" podID="4ae999a1-2c38-459d-bf86-e4928c78a668" containerID="e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd" exitCode=0
Sep 09 20:46:21.017878 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.017800    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"
Sep 09 20:46:21.017878 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.017847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd" event={"ID":"4ae999a1-2c38-459d-bf86-e4928c78a668","Type":"ContainerDied","Data":"e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd"}
Sep 09 20:46:21.017991 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.017898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd" event={"ID":"4ae999a1-2c38-459d-bf86-e4928c78a668","Type":"ContainerDied","Data":"4bc54d5defbd03a925d1c80970300837cd872ad7ae36f593cd91d580448215d4"}
Sep 09 20:46:21.027816 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.027793    2581 scope.go:117] "RemoveContainer" containerID="bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50"
Sep 09 20:46:21.028272 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:46:21.028250    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50\": container with ID starting with bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50 not found: ID does not exist" containerID="bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50"
Sep 09 20:46:21.028401 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.028284    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50"} err="failed to get container status \"bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50\": rpc error: code = NotFound desc = could not find container \"bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50\": container with ID starting with bfbafcec5784413df72bad9d25cc372d3b41be0c8198f91260e38cab41143d50 not found: ID does not exist"
Sep 09 20:46:21.028401 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.028307    2581 scope.go:117] "RemoveContainer" containerID="e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd"
Sep 09 20:46:21.036799 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.036782    2581 scope.go:117] "RemoveContainer" containerID="e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd"
Sep 09 20:46:21.037067 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:46:21.037048    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd\": container with ID starting with e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd not found: ID does not exist" containerID="e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd"
Sep 09 20:46:21.037132 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.037077    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd"} err="failed to get container status \"e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd\": rpc error: code = NotFound desc = could not find container \"e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd\": container with ID starting with e0623bf5ecd0c8c6a33e66d1450829607479db9e8a896e3426a4e46a06245bbd not found: ID does not exist"
Sep 09 20:46:21.064608 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.064583    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"]
Sep 09 20:46:21.077541 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.077519    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-697f19-75ccbbbf5-6rkqd"]
Sep 09 20:46:21.128472 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.128436    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"]
Sep 09 20:46:21.148295 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:21.148263    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-697f19-5cff56b958-ks67g"]
Sep 09 20:46:23.002451 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:23.002409    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ae999a1-2c38-459d-bf86-e4928c78a668" path="/var/lib/kubelet/pods/4ae999a1-2c38-459d-bf86-e4928c78a668/volumes"
Sep 09 20:46:23.003046 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:23.002967    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c39a7452-c585-4949-852b-5655ef355270" path="/var/lib/kubelet/pods/c39a7452-c585-4949-852b-5655ef355270/volumes"
Sep 09 20:46:40.169040 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:40.169006    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w"]
Sep 09 20:46:40.174874 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:40.174852    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-166b94-29816445-hnw2w"]
Sep 09 20:46:40.245129 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:40.245093    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"]
Sep 09 20:46:40.245392 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:40.245362    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f" podUID="1bd19932-cb24-4c3e-9e4f-9d956d76a89d" containerName="payload-pre-processing" containerID="cri-o://ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c" gracePeriod=30
Sep 09 20:46:40.506075 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:40.506050    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:46:40.599577 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:40.599549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhrnk\" (UniqueName: \"kubernetes.io/projected/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-kube-api-access-hhrnk\") pod \"1bd19932-cb24-4c3e-9e4f-9d956d76a89d\" (UID: \"1bd19932-cb24-4c3e-9e4f-9d956d76a89d\") "
Sep 09 20:46:40.599739 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:40.599630    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-plugins-config-volume\") pod \"1bd19932-cb24-4c3e-9e4f-9d956d76a89d\" (UID: \"1bd19932-cb24-4c3e-9e4f-9d956d76a89d\") "
Sep 09 20:46:40.599989 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:40.599968    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1bd19932-cb24-4c3e-9e4f-9d956d76a89d" (UID: "1bd19932-cb24-4c3e-9e4f-9d956d76a89d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:46:40.601829 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:40.601803    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-kube-api-access-hhrnk" (OuterVolumeSpecName: "kube-api-access-hhrnk") pod "1bd19932-cb24-4c3e-9e4f-9d956d76a89d" (UID: "1bd19932-cb24-4c3e-9e4f-9d956d76a89d"). InnerVolumeSpecName "kube-api-access-hhrnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:46:40.701162 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:40.701123    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhrnk\" (UniqueName: \"kubernetes.io/projected/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-kube-api-access-hhrnk\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:46:40.701162 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:40.701156    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bd19932-cb24-4c3e-9e4f-9d956d76a89d-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:46:41.002493 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:41.002457    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cfb7581-8ecc-40bb-9d38-d5085d419b00" path="/var/lib/kubelet/pods/5cfb7581-8ecc-40bb-9d38-d5085d419b00/volumes"
Sep 09 20:46:41.131073 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:41.131036    2581 generic.go:356] "Generic (PLEG): container finished" podID="1bd19932-cb24-4c3e-9e4f-9d956d76a89d" containerID="ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c" exitCode=0
Sep 09 20:46:41.131236 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:41.131091    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"
Sep 09 20:46:41.131236 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:41.131132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f" event={"ID":"1bd19932-cb24-4c3e-9e4f-9d956d76a89d","Type":"ContainerDied","Data":"ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c"}
Sep 09 20:46:41.131236 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:41.131183    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f" event={"ID":"1bd19932-cb24-4c3e-9e4f-9d956d76a89d","Type":"ContainerDied","Data":"c10a71d4e561fbf6c4eaad41d1c14af83ff71b5debfe09f443c7303c7c74fe2b"}
Sep 09 20:46:41.131236 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:41.131211    2581 scope.go:117] "RemoveContainer" containerID="ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c"
Sep 09 20:46:41.140133 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:41.140116    2581 scope.go:117] "RemoveContainer" containerID="ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c"
Sep 09 20:46:41.140426 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:46:41.140407    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c\": container with ID starting with ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c not found: ID does not exist" containerID="ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c"
Sep 09 20:46:41.140473 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:41.140436    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c"} err="failed to get container status \"ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c\": rpc error: code = NotFound desc = could not find container \"ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c\": container with ID starting with ccfcd5cabe6bc6a21568e62cc5b24b65bdb5cc97915b91371f1b17c3ef828a7c not found: ID does not exist"
Sep 09 20:46:41.173078 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:41.173048    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"]
Sep 09 20:46:41.223823 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:41.223792    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-166b94-77f5bccd78-t7f5f"]
Sep 09 20:46:43.002453 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:46:43.002420    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bd19932-cb24-4c3e-9e4f-9d956d76a89d" path="/var/lib/kubelet/pods/1bd19932-cb24-4c3e-9e4f-9d956d76a89d/volumes"
Sep 09 20:47:01.265004 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.264966    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"]
Sep 09 20:47:01.265498 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.265189    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x" podUID="5ae35504-f913-4961-8764-b19dd9c9e7e9" containerName="payload-processing" containerID="cri-o://ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4" gracePeriod=30
Sep 09 20:47:01.401392 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.401331    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"]
Sep 09 20:47:01.401674 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.401643    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp" podUID="0cc92021-806a-46b8-bdb2-8a5b40431f02" containerName="payload-pre-processing" containerID="cri-o://c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c" gracePeriod=30
Sep 09 20:47:01.522643 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.522619    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:47:01.593882 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.593853    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzszv\" (UniqueName: \"kubernetes.io/projected/5ae35504-f913-4961-8764-b19dd9c9e7e9-kube-api-access-nzszv\") pod \"5ae35504-f913-4961-8764-b19dd9c9e7e9\" (UID: \"5ae35504-f913-4961-8764-b19dd9c9e7e9\") "
Sep 09 20:47:01.594068 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.593943    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae35504-f913-4961-8764-b19dd9c9e7e9-plugins-config-volume\") pod \"5ae35504-f913-4961-8764-b19dd9c9e7e9\" (UID: \"5ae35504-f913-4961-8764-b19dd9c9e7e9\") "
Sep 09 20:47:01.594440 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.594398    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae35504-f913-4961-8764-b19dd9c9e7e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ae35504-f913-4961-8764-b19dd9c9e7e9" (UID: "5ae35504-f913-4961-8764-b19dd9c9e7e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:01.596184 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.596154    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae35504-f913-4961-8764-b19dd9c9e7e9-kube-api-access-nzszv" (OuterVolumeSpecName: "kube-api-access-nzszv") pod "5ae35504-f913-4961-8764-b19dd9c9e7e9" (UID: "5ae35504-f913-4961-8764-b19dd9c9e7e9"). InnerVolumeSpecName "kube-api-access-nzszv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:01.639075 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.639041    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:47:01.695495 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.695462    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzszv\" (UniqueName: \"kubernetes.io/projected/5ae35504-f913-4961-8764-b19dd9c9e7e9-kube-api-access-nzszv\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:47:01.695495 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.695496    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae35504-f913-4961-8764-b19dd9c9e7e9-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:47:01.795868 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.795783    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc92021-806a-46b8-bdb2-8a5b40431f02-plugins-config-volume\") pod \"0cc92021-806a-46b8-bdb2-8a5b40431f02\" (UID: \"0cc92021-806a-46b8-bdb2-8a5b40431f02\") "
Sep 09 20:47:01.795868 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.795830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vkgv\" (UniqueName: \"kubernetes.io/projected/0cc92021-806a-46b8-bdb2-8a5b40431f02-kube-api-access-4vkgv\") pod \"0cc92021-806a-46b8-bdb2-8a5b40431f02\" (UID: \"0cc92021-806a-46b8-bdb2-8a5b40431f02\") "
Sep 09 20:47:01.796159 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.796136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc92021-806a-46b8-bdb2-8a5b40431f02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0cc92021-806a-46b8-bdb2-8a5b40431f02" (UID: "0cc92021-806a-46b8-bdb2-8a5b40431f02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:01.798095 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.798074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc92021-806a-46b8-bdb2-8a5b40431f02-kube-api-access-4vkgv" (OuterVolumeSpecName: "kube-api-access-4vkgv") pod "0cc92021-806a-46b8-bdb2-8a5b40431f02" (UID: "0cc92021-806a-46b8-bdb2-8a5b40431f02"). InnerVolumeSpecName "kube-api-access-4vkgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:01.897100 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.897058    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cc92021-806a-46b8-bdb2-8a5b40431f02-plugins-config-volume\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:47:01.897100 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:01.897090    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vkgv\" (UniqueName: \"kubernetes.io/projected/0cc92021-806a-46b8-bdb2-8a5b40431f02-kube-api-access-4vkgv\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:47:02.249583 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.249549    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ae35504-f913-4961-8764-b19dd9c9e7e9" containerID="ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4" exitCode=0
Sep 09 20:47:02.249791 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.249600    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"
Sep 09 20:47:02.249791 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.249638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x" event={"ID":"5ae35504-f913-4961-8764-b19dd9c9e7e9","Type":"ContainerDied","Data":"ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4"}
Sep 09 20:47:02.249791 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.249699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x" event={"ID":"5ae35504-f913-4961-8764-b19dd9c9e7e9","Type":"ContainerDied","Data":"bd01dd3069e48885bec3eaa9ef98be86e2230c5e052ad1402d7ced8d9441e8df"}
Sep 09 20:47:02.249791 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.249728    2581 scope.go:117] "RemoveContainer" containerID="ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4"
Sep 09 20:47:02.251045 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.251020    2581 generic.go:356] "Generic (PLEG): container finished" podID="0cc92021-806a-46b8-bdb2-8a5b40431f02" containerID="c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c" exitCode=0
Sep 09 20:47:02.251149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.251055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp" event={"ID":"0cc92021-806a-46b8-bdb2-8a5b40431f02","Type":"ContainerDied","Data":"c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c"}
Sep 09 20:47:02.251149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.251069    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"
Sep 09 20:47:02.251149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.251121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp" event={"ID":"0cc92021-806a-46b8-bdb2-8a5b40431f02","Type":"ContainerDied","Data":"e9625d14dfbf01ff48dda8248db62524c85da963fec5bcc7d99fa85bb63dc767"}
Sep 09 20:47:02.259136 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.259115    2581 scope.go:117] "RemoveContainer" containerID="ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4"
Sep 09 20:47:02.259589 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:47:02.259565    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4\": container with ID starting with ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4 not found: ID does not exist" containerID="ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4"
Sep 09 20:47:02.259702 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.259596    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4"} err="failed to get container status \"ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4\": rpc error: code = NotFound desc = could not find container \"ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4\": container with ID starting with ba3f967eb848c112ac2226da543cef4feb5fc6c879681d9d6cf8937a6fd771d4 not found: ID does not exist"
Sep 09 20:47:02.259702 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.259614    2581 scope.go:117] "RemoveContainer" containerID="c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c"
Sep 09 20:47:02.268418 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.268180    2581 scope.go:117] "RemoveContainer" containerID="c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c"
Sep 09 20:47:02.268678 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:47:02.268507    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c\": container with ID starting with c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c not found: ID does not exist" containerID="c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c"
Sep 09 20:47:02.268678 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.268537    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c"} err="failed to get container status \"c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c\": rpc error: code = NotFound desc = could not find container \"c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c\": container with ID starting with c7531e37f2aa92aa4879f1b2a8bf44783297b9be78fd788ac9bf47c9a497bb4c not found: ID does not exist"
Sep 09 20:47:02.330355 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.330311    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"]
Sep 09 20:47:02.338931 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.338905    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4101f2-6b58849f97-ttw8x"]
Sep 09 20:47:02.384457 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.384424    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"]
Sep 09 20:47:02.432156 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:02.432113    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4101f2-79d5f69998-67znp"]
Sep 09 20:47:03.002426 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:03.002390    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cc92021-806a-46b8-bdb2-8a5b40431f02" path="/var/lib/kubelet/pods/0cc92021-806a-46b8-bdb2-8a5b40431f02/volumes"
Sep 09 20:47:03.002763 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:03.002750    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae35504-f913-4961-8764-b19dd9c9e7e9" path="/var/lib/kubelet/pods/5ae35504-f913-4961-8764-b19dd9c9e7e9/volumes"
Sep 09 20:47:19.444394 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:19.444360    2581 scope.go:117] "RemoveContainer" containerID="6f9544fc64b15089ddc622a2a45e60d6306455184a00c34687ae32eba750e31c"
Sep 09 20:47:19.453089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:47:19.453064    2581 scope.go:117] "RemoveContainer" containerID="5112a7faf2d204fb290c9adc3e808801ec2f2284ba2fa5e2d57dc35567033374"
Sep 09 20:48:19.497385 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:48:19.497324    2581 scope.go:117] "RemoveContainer" containerID="c9047e552dd088bccbf7c49fc2694e0048b3eb1f2ae37e1de37df9444dc47318"
Sep 09 20:51:19.155714 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:19.155687    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:51:19.160228 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:19.160205    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:51:19.534660 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:19.534623    2581 scope.go:117] "RemoveContainer" containerID="328d49a91c32430e0d51042e8908e97cf5f4c565b2e006691e7f2093e1dfa194"
Sep 09 20:51:19.543852 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:19.543830    2581 scope.go:117] "RemoveContainer" containerID="82dfb53bd3b69cd911e5e596887ebbe96f9b82205ef3f9ffd8df21978d8fbffa"
Sep 09 20:51:19.551730 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:19.551712    2581 scope.go:117] "RemoveContainer" containerID="642de41c8c3bc0e223dc2b053ad1569c57592bf984abf29cb96baf1556effb8e"
Sep 09 20:51:36.266531 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.266495    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"]
Sep 09 20:51:36.267001 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.266985    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cc92021-806a-46b8-bdb2-8a5b40431f02" containerName="payload-pre-processing"
Sep 09 20:51:36.267052 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267003    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cc92021-806a-46b8-bdb2-8a5b40431f02" containerName="payload-pre-processing"
Sep 09 20:51:36.267052 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267012    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae35504-f913-4961-8764-b19dd9c9e7e9" containerName="payload-processing"
Sep 09 20:51:36.267052 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267018    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae35504-f913-4961-8764-b19dd9c9e7e9" containerName="payload-processing"
Sep 09 20:51:36.267052 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267035    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae999a1-2c38-459d-bf86-e4928c78a668" containerName="payload-processing"
Sep 09 20:51:36.267052 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267040    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae999a1-2c38-459d-bf86-e4928c78a668" containerName="payload-processing"
Sep 09 20:51:36.267052 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267047    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c39a7452-c585-4949-852b-5655ef355270" containerName="payload-pre-processing"
Sep 09 20:51:36.267052 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267053    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c39a7452-c585-4949-852b-5655ef355270" containerName="payload-pre-processing"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267065    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5592a6-debc-46c5-b7dd-88d37de23ad7" containerName="main"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267070    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5592a6-debc-46c5-b7dd-88d37de23ad7" containerName="main"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267081    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd7eab20-2a51-477d-bfe3-39e86975b107" containerName="payload-pre-processing"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267086    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd7eab20-2a51-477d-bfe3-39e86975b107" containerName="payload-pre-processing"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267091    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99502b67-71fc-4dfd-9f61-924adb4317fa" containerName="revoke-keys"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267096    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="99502b67-71fc-4dfd-9f61-924adb4317fa" containerName="revoke-keys"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267107    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd19932-cb24-4c3e-9e4f-9d956d76a89d" containerName="payload-pre-processing"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267113    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd19932-cb24-4c3e-9e4f-9d956d76a89d" containerName="payload-pre-processing"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267168    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae999a1-2c38-459d-bf86-e4928c78a668" containerName="payload-processing"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267177    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5592a6-debc-46c5-b7dd-88d37de23ad7" containerName="main"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267185    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c39a7452-c585-4949-852b-5655ef355270" containerName="payload-pre-processing"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267189    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bd19932-cb24-4c3e-9e4f-9d956d76a89d" containerName="payload-pre-processing"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267195    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae35504-f913-4961-8764-b19dd9c9e7e9" containerName="payload-processing"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267201    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd7eab20-2a51-477d-bfe3-39e86975b107" containerName="payload-pre-processing"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267206    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cc92021-806a-46b8-bdb2-8a5b40431f02" containerName="payload-pre-processing"
Sep 09 20:51:36.267300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.267211    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="99502b67-71fc-4dfd-9f61-924adb4317fa" containerName="revoke-keys"
Sep 09 20:51:36.270185 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.270167    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.272962 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.272938    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3d9d523f-kserve-self-signed-certs\""
Sep 09 20:51:36.280737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.280714    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"]
Sep 09 20:51:36.367311 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.367278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-model-cache\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.367311 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.367318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-home\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.367548 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.367354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tmp-dir\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.367548 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.367478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tls-certs\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.367548 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.367513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-dshm\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.367651 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.367560    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9whg7\" (UniqueName: \"kubernetes.io/projected/8e2ee4a3-a73c-40d0-862a-bcf92da04431-kube-api-access-9whg7\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.468534 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.468496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-model-cache\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.468534 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.468541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-home\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.468788 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.468565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tmp-dir\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.468788 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.468644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tls-certs\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.468788 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.468675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-dshm\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.468788 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.468706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9whg7\" (UniqueName: \"kubernetes.io/projected/8e2ee4a3-a73c-40d0-862a-bcf92da04431-kube-api-access-9whg7\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.468999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.468907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-home\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.468999 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.468972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-model-cache\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.469129 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.469007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tmp-dir\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.471158 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.471131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-dshm\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.471358 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.471323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tls-certs\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.476686 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.476662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9whg7\" (UniqueName: \"kubernetes.io/projected/8e2ee4a3-a73c-40d0-862a-bcf92da04431-kube-api-access-9whg7\") pod \"e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") " pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.580510 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.580411    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:36.859602 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:36.859572    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"]
Sep 09 20:51:36.861313 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:51:36.861285    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e2ee4a3_a73c_40d0_862a_bcf92da04431.slice/crio-4f7f094b593632fc26df3ab80c387d49282a04d36d9c0751e5c53e41f9cf3291 WatchSource:0}: Error finding container 4f7f094b593632fc26df3ab80c387d49282a04d36d9c0751e5c53e41f9cf3291: Status 404 returned error can't find the container with id 4f7f094b593632fc26df3ab80c387d49282a04d36d9c0751e5c53e41f9cf3291
Sep 09 20:51:37.724227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:37.724187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65" event={"ID":"8e2ee4a3-a73c-40d0-862a-bcf92da04431","Type":"ContainerStarted","Data":"129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10"}
Sep 09 20:51:37.724227 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:37.724235    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65" event={"ID":"8e2ee4a3-a73c-40d0-862a-bcf92da04431","Type":"ContainerStarted","Data":"4f7f094b593632fc26df3ab80c387d49282a04d36d9c0751e5c53e41f9cf3291"}
Sep 09 20:51:37.743246 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:37.743197    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65" podStartSLOduration=1.743180927 podStartE2EDuration="1.743180927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:37.741540934 +0000 UTC m=+2119.234751916" watchObservedRunningTime="2026-09-09 20:51:37.743180927 +0000 UTC m=+2119.236391910"
Sep 09 20:51:46.581046 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:46.581004    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:46.581538 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:46.581089    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:46.594096 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:46.594072    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:51:46.783567 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:51:46.783530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:52:38.567773 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:38.567731    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"]
Sep 09 20:52:53.587868 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.587830    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65" podUID="8e2ee4a3-a73c-40d0-862a-bcf92da04431" containerName="main" containerID="cri-o://129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10" gracePeriod=45
Sep 09 20:52:53.841321 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.841258    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:52:53.953737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.953701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-dshm\") pod \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") "
Sep 09 20:52:53.953737 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.953743    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-model-cache\") pod \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") "
Sep 09 20:52:53.954089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.953791    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-home\") pod \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") "
Sep 09 20:52:53.954089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.953823    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9whg7\" (UniqueName: \"kubernetes.io/projected/8e2ee4a3-a73c-40d0-862a-bcf92da04431-kube-api-access-9whg7\") pod \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") "
Sep 09 20:52:53.954089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.953875    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tls-certs\") pod \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") "
Sep 09 20:52:53.954089 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.953907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tmp-dir\") pod \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\" (UID: \"8e2ee4a3-a73c-40d0-862a-bcf92da04431\") "
Sep 09 20:52:53.954307 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.954132    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-home" (OuterVolumeSpecName: "home") pod "8e2ee4a3-a73c-40d0-862a-bcf92da04431" (UID: "8e2ee4a3-a73c-40d0-862a-bcf92da04431"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:53.954307 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.954149    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e2ee4a3-a73c-40d0-862a-bcf92da04431" (UID: "8e2ee4a3-a73c-40d0-862a-bcf92da04431"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:53.954307 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.954284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e2ee4a3-a73c-40d0-862a-bcf92da04431" (UID: "8e2ee4a3-a73c-40d0-862a-bcf92da04431"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:53.956264 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.956238    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-dshm" (OuterVolumeSpecName: "dshm") pod "8e2ee4a3-a73c-40d0-862a-bcf92da04431" (UID: "8e2ee4a3-a73c-40d0-862a-bcf92da04431"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:52:53.956264 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.956239    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e2ee4a3-a73c-40d0-862a-bcf92da04431" (UID: "8e2ee4a3-a73c-40d0-862a-bcf92da04431"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:52:53.956408 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:53.956237    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2ee4a3-a73c-40d0-862a-bcf92da04431-kube-api-access-9whg7" (OuterVolumeSpecName: "kube-api-access-9whg7") pod "8e2ee4a3-a73c-40d0-862a-bcf92da04431" (UID: "8e2ee4a3-a73c-40d0-862a-bcf92da04431"). InnerVolumeSpecName "kube-api-access-9whg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:52:54.055244 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.055206    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-dshm\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:52:54.055244 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.055238    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-model-cache\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:52:54.055244 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.055247    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-home\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:52:54.055523 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.055256    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9whg7\" (UniqueName: \"kubernetes.io/projected/8e2ee4a3-a73c-40d0-862a-bcf92da04431-kube-api-access-9whg7\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:52:54.055523 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.055265    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tls-certs\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:52:54.055523 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.055275    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2ee4a3-a73c-40d0-862a-bcf92da04431-tmp-dir\") on node \"ip-10-0-135-122.ec2.internal\" DevicePath \"\""
Sep 09 20:52:54.128398 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.128283    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e2ee4a3-a73c-40d0-862a-bcf92da04431" containerID="129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10" exitCode=0
Sep 09 20:52:54.128398 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.128330    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"
Sep 09 20:52:54.128398 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.128363    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65" event={"ID":"8e2ee4a3-a73c-40d0-862a-bcf92da04431","Type":"ContainerDied","Data":"129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10"}
Sep 09 20:52:54.128398 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.128395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65" event={"ID":"8e2ee4a3-a73c-40d0-862a-bcf92da04431","Type":"ContainerDied","Data":"4f7f094b593632fc26df3ab80c387d49282a04d36d9c0751e5c53e41f9cf3291"}
Sep 09 20:52:54.128690 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.128412    2581 scope.go:117] "RemoveContainer" containerID="129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10"
Sep 09 20:52:54.138111 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.138093    2581 scope.go:117] "RemoveContainer" containerID="129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10"
Sep 09 20:52:54.138428 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:52:54.138402    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10\": container with ID starting with 129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10 not found: ID does not exist" containerID="129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10"
Sep 09 20:52:54.138495 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.138441    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10"} err="failed to get container status \"129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10\": rpc error: code = NotFound desc = could not find container \"129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10\": container with ID starting with 129ddd8fdf4a3d295f1d1ed7de200387dbd6a2984f0ba8460eec9ef34dd8fa10 not found: ID does not exist"
Sep 09 20:52:54.155150 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.155125    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"]
Sep 09 20:52:54.158802 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:54.158774    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3d9d523f-kserve-6fd58d9479-67n65"]
Sep 09 20:52:55.003432 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:52:55.003399    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e2ee4a3-a73c-40d0-862a-bcf92da04431" path="/var/lib/kubelet/pods/8e2ee4a3-a73c-40d0-862a-bcf92da04431/volumes"
Sep 09 20:53:38.645717 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:38.645679    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69b688bd6c-s297w_308493ad-7d66-45a1-8b11-828830f7b3a1/authorino/0.log"
Sep 09 20:53:45.004449 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:45.004358    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n_3d171978-73e5-4a61-9eaf-8ff6047520fb/copy-manifests/0.log"
Sep 09 20:53:45.030133 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:45.030105    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n_3d171978-73e5-4a61-9eaf-8ff6047520fb/manager/0.log"
Sep 09 20:53:46.797938 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:46.797907    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69b688bd6c-s297w_308493ad-7d66-45a1-8b11-828830f7b3a1/authorino/0.log"
Sep 09 20:53:47.202735 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:47.202711    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-zmhwn_8aeef901-a7d1-49e2-bab8-0c578fb19e1c/kuadrant-console-plugin/0.log"
Sep 09 20:53:48.202488 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:48.202454    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758cfc744b-627r6_d14e73dc-c91b-420c-8d6e-d59c9bb38ff0/kube-auth-proxy/0.log"
Sep 09 20:53:48.562611 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:48.562538    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qzrnd_0e70b798-2892-471a-bf91-81797e3ff923/payload-pre-processing/0.log"
Sep 09 20:53:48.683156 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:48.683128    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zdk65_63bec830-c2e4-4a4a-a998-280cbae1baf7/payload-processing/0.log"
Sep 09 20:53:48.806028 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:48.806000    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c4444dc76-c2dcn_23891ba7-4710-4468-b69d-8764b3cf8384/router/0.log"
Sep 09 20:53:49.480887 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:49.480852    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-xwx4v_f1c7f0ec-985b-4045-8d39-c2682fc2a2cb/main/0.log"
Sep 09 20:53:49.715502 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:49.715472    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-5q5tk_02255117-9878-4ac7-9f9b-5fc375a7cdab/main/0.log"
Sep 09 20:53:49.834518 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:49.834445    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-h577s_8e634968-5d7e-42ec-ac1f-efe0037f1527/main/0.log"
Sep 09 20:53:55.731686 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:55.731649    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mp9c5/must-gather-mz7n4"]
Sep 09 20:53:55.732141 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:55.732003    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e2ee4a3-a73c-40d0-862a-bcf92da04431" containerName="main"
Sep 09 20:53:55.732141 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:55.732014    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e2ee4a3-a73c-40d0-862a-bcf92da04431" containerName="main"
Sep 09 20:53:55.732141 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:55.732073    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e2ee4a3-a73c-40d0-862a-bcf92da04431" containerName="main"
Sep 09 20:53:55.734997 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:55.734980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mp9c5/must-gather-mz7n4"
Sep 09 20:53:55.740451 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:53:55.740425    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-j5dnh\" is forbidden: User \"system:node:ip-10-0-135-122.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-must-gather-mp9c5\": no relationship found between node 'ip-10-0-135-122.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-mp9c5\"/\"default-dockercfg-j5dnh\"" type="*v1.Secret"
Sep 09 20:53:55.740784 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:53:55.740760    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-122.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-must-gather-mp9c5\": no relationship found between node 'ip-10-0-135-122.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-mp9c5\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 09 20:53:55.740901 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:53:55.740767    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-122.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-must-gather-mp9c5\": no relationship found between node 'ip-10-0-135-122.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-must-gather-mp9c5\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 09 20:53:55.740901 ip-10-0-135-122 kubenswrapper[2581]: E0909 20:53:55.740859    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-mz7n4\" is forbidden: User \"system:node:ip-10-0-135-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mp9c5\": no relationship found between node 'ip-10-0-135-122.ec2.internal' and this object" podUID="3d1b3493-7ba8-444b-9cbc-bd053e51ffbc" pod="openshift-must-gather-mp9c5/must-gather-mz7n4"
Sep 09 20:53:55.789332 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:55.789305    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mp9c5/must-gather-mz7n4"]
Sep 09 20:53:55.880308 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:55.880269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq727\" (UniqueName: \"kubernetes.io/projected/3d1b3493-7ba8-444b-9cbc-bd053e51ffbc-kube-api-access-vq727\") pod \"must-gather-mz7n4\" (UID: \"3d1b3493-7ba8-444b-9cbc-bd053e51ffbc\") " pod="openshift-must-gather-mp9c5/must-gather-mz7n4"
Sep 09 20:53:55.880508 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:55.880441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d1b3493-7ba8-444b-9cbc-bd053e51ffbc-must-gather-output\") pod \"must-gather-mz7n4\" (UID: \"3d1b3493-7ba8-444b-9cbc-bd053e51ffbc\") " pod="openshift-must-gather-mp9c5/must-gather-mz7n4"
Sep 09 20:53:55.981587 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:55.981554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d1b3493-7ba8-444b-9cbc-bd053e51ffbc-must-gather-output\") pod \"must-gather-mz7n4\" (UID: \"3d1b3493-7ba8-444b-9cbc-bd053e51ffbc\") " pod="openshift-must-gather-mp9c5/must-gather-mz7n4"
Sep 09 20:53:55.981761 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:55.981597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq727\" (UniqueName: \"kubernetes.io/projected/3d1b3493-7ba8-444b-9cbc-bd053e51ffbc-kube-api-access-vq727\") pod \"must-gather-mz7n4\" (UID: \"3d1b3493-7ba8-444b-9cbc-bd053e51ffbc\") " pod="openshift-must-gather-mp9c5/must-gather-mz7n4"
Sep 09 20:53:55.981902 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:55.981881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3d1b3493-7ba8-444b-9cbc-bd053e51ffbc-must-gather-output\") pod \"must-gather-mz7n4\" (UID: \"3d1b3493-7ba8-444b-9cbc-bd053e51ffbc\") " pod="openshift-must-gather-mp9c5/must-gather-mz7n4"
Sep 09 20:53:56.624893 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:56.624865    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mp9c5\"/\"kube-root-ca.crt\""
Sep 09 20:53:56.663038 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:56.663005    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mp9c5\"/\"openshift-service-ca.crt\""
Sep 09 20:53:56.669385 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:56.669358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq727\" (UniqueName: \"kubernetes.io/projected/3d1b3493-7ba8-444b-9cbc-bd053e51ffbc-kube-api-access-vq727\") pod \"must-gather-mz7n4\" (UID: \"3d1b3493-7ba8-444b-9cbc-bd053e51ffbc\") " pod="openshift-must-gather-mp9c5/must-gather-mz7n4"
Sep 09 20:53:57.289145 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:57.289102    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mp9c5\"/\"default-dockercfg-j5dnh\""
Sep 09 20:53:57.294883 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:57.294860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mp9c5/must-gather-mz7n4"
Sep 09 20:53:57.576243 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:57.576216    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mp9c5/must-gather-mz7n4"]
Sep 09 20:53:57.576915 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:53:57.576892    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d1b3493_7ba8_444b_9cbc_bd053e51ffbc.slice/crio-756e78750c840c32b0c1b3b93dbd298af7aa1468948ff5230ce828929d05ae64 WatchSource:0}: Error finding container 756e78750c840c32b0c1b3b93dbd298af7aa1468948ff5230ce828929d05ae64: Status 404 returned error can't find the container with id 756e78750c840c32b0c1b3b93dbd298af7aa1468948ff5230ce828929d05ae64
Sep 09 20:53:57.578824 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:57.578807    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:53:58.470752 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:58.470710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mp9c5/must-gather-mz7n4" event={"ID":"3d1b3493-7ba8-444b-9cbc-bd053e51ffbc","Type":"ContainerStarted","Data":"756e78750c840c32b0c1b3b93dbd298af7aa1468948ff5230ce828929d05ae64"}
Sep 09 20:53:59.479750 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:59.479713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mp9c5/must-gather-mz7n4" event={"ID":"3d1b3493-7ba8-444b-9cbc-bd053e51ffbc","Type":"ContainerStarted","Data":"ff680b8f8befcc6cc960e359f186a1b919752a1a61b9eca42fefec244ca49d99"}
Sep 09 20:53:59.479750 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:59.479756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mp9c5/must-gather-mz7n4" event={"ID":"3d1b3493-7ba8-444b-9cbc-bd053e51ffbc","Type":"ContainerStarted","Data":"726617928a061ba691969da6c17128c5f1c9d26b61a70a93b6b2c7cff1259923"}
Sep 09 20:53:59.514923 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:53:59.514860    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mp9c5/must-gather-mz7n4" podStartSLOduration=3.671555261 podStartE2EDuration="4.514838223s" totalImagesPullingTime="843.282962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:59.510010913 +0000 UTC m=+2261.003221897" watchObservedRunningTime="2026-09-09 20:53:59.514838223 +0000 UTC m=+2261.008049210"
Sep 09 20:54:00.421248 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:00.421214    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2f9sh_3774603a-6336-4e25-82a3-c5a0b89fd082/global-pull-secret-syncer/0.log"
Sep 09 20:54:01.078900 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:01.078870    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sdghr_89368c79-9c74-4509-8588-0600a7e46763/konnectivity-agent/0.log"
Sep 09 20:54:01.160678 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:01.160650    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-122.ec2.internal_2f1edbe788da0ad39b612586be095693/haproxy/0.log"
Sep 09 20:54:05.172149 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:05.172113    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69b688bd6c-s297w_308493ad-7d66-45a1-8b11-828830f7b3a1/authorino/0.log"
Sep 09 20:54:05.402830 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:05.402796    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-zmhwn_8aeef901-a7d1-49e2-bab8-0c578fb19e1c/kuadrant-console-plugin/0.log"
Sep 09 20:54:07.273320 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:07.273274    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-mfr42_cc4d0a36-bc84-4ffe-b5d8-804d35114373/cluster-monitoring-operator/0.log"
Sep 09 20:54:07.925031 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:07.924998    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2h7c_a73d7050-de36-4448-8292-9655e23eff84/node-exporter/0.log"
Sep 09 20:54:07.955756 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:07.955724    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2h7c_a73d7050-de36-4448-8292-9655e23eff84/kube-rbac-proxy/0.log"
Sep 09 20:54:07.981071 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:07.981045    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v2h7c_a73d7050-de36-4448-8292-9655e23eff84/init-textfile/0.log"
Sep 09 20:54:09.989642 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:09.989612    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-mlwpc_e937672f-c395-458f-890d-aea135850301/networking-console-plugin/0.log"
Sep 09 20:54:10.069775 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.069738    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"]
Sep 09 20:54:10.076145 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.076117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.084331 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.084301    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"]
Sep 09 20:54:10.132809 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.132772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-lib-modules\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.133103 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.133078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-proc\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.133282 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.133264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-podres\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.133475 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.133457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-sys\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.133653 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.133633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf8jv\" (UniqueName: \"kubernetes.io/projected/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-kube-api-access-pf8jv\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.234170 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.234134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-proc\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.234392 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.234184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-podres\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.234392 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.234238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-sys\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.234392 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.234277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf8jv\" (UniqueName: \"kubernetes.io/projected/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-kube-api-access-pf8jv\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.234392 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.234284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-proc\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.234392 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.234377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-lib-modules\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.234759 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.234422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-podres\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.234759 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.234496    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-sys\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.234759 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.234530    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-lib-modules\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.246884 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.246815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf8jv\" (UniqueName: \"kubernetes.io/projected/eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc-kube-api-access-pf8jv\") pod \"perf-node-gather-daemonset-klp5g\" (UID: \"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc\") " pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.391487 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.391448    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:10.675618 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.675516    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/2.log"
Sep 09 20:54:10.682300 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.682274    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-7c664_e3ef9dbb-ce20-4e5c-9eb4-5f94a7c2cf68/console-operator/3.log"
Sep 09 20:54:10.716923 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:10.716872    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"]
Sep 09 20:54:10.717926 ip-10-0-135-122 kubenswrapper[2581]: W0909 20:54:10.717898    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeb78d48a_9e31_4be8_bd7a_a3a0f2288fcc.slice/crio-827095d945403d1ebfc321e9049f1605ea702f9487b9a75d6ac77a58bb2b95c8 WatchSource:0}: Error finding container 827095d945403d1ebfc321e9049f1605ea702f9487b9a75d6ac77a58bb2b95c8: Status 404 returned error can't find the container with id 827095d945403d1ebfc321e9049f1605ea702f9487b9a75d6ac77a58bb2b95c8
Sep 09 20:54:11.272914 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:11.272878    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c78ccbc66-dlwks_1c9a21e8-23c7-4e4c-92ab-bb91f419dbea/console/0.log"
Sep 09 20:54:11.583411 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:11.583297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g" event={"ID":"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc","Type":"ContainerStarted","Data":"852f1180e8ec2eeacd4b568823a332aecbe053dabdf779242280317425537204"}
Sep 09 20:54:11.583411 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:11.583391    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:11.583411 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:11.583412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g" event={"ID":"eb78d48a-9e31-4be8-bd7a-a3a0f2288fcc","Type":"ContainerStarted","Data":"827095d945403d1ebfc321e9049f1605ea702f9487b9a75d6ac77a58bb2b95c8"}
Sep 09 20:54:11.607753 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:11.607683    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g" podStartSLOduration=1.607663913 podStartE2EDuration="1.607663913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:11.606106709 +0000 UTC m=+2273.099317694" watchObservedRunningTime="2026-09-09 20:54:11.607663913 +0000 UTC m=+2273.100874898"
Sep 09 20:54:12.081134 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:12.081096    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-c57sd_2be67041-c131-4335-a192-7476a5fcdbf1/volume-data-source-validator/0.log"
Sep 09 20:54:13.305293 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:13.305200    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z7wr5_48d0e306-fdd9-4e95-ab7c-5ebdbb06f444/dns/0.log"
Sep 09 20:54:13.329041 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:13.329010    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z7wr5_48d0e306-fdd9-4e95-ab7c-5ebdbb06f444/kube-rbac-proxy/0.log"
Sep 09 20:54:13.353156 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:13.353131    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2xswt_e5088b6b-85b2-4c2f-b7b6-0f1039ba191a/dns-node-resolver/0.log"
Sep 09 20:54:14.004319 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:14.004284    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2859f_83456891-bf2f-4d7d-82a6-cb2820c78575/node-ca/0.log"
Sep 09 20:54:15.664834 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:15.664798    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758cfc744b-627r6_d14e73dc-c91b-420c-8d6e-d59c9bb38ff0/kube-auth-proxy/0.log"
Sep 09 20:54:15.774987 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:15.774958    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qzrnd_0e70b798-2892-471a-bf91-81797e3ff923/payload-pre-processing/0.log"
Sep 09 20:54:15.815539 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:15.815503    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zdk65_63bec830-c2e4-4a4a-a998-280cbae1baf7/payload-processing/0.log"
Sep 09 20:54:15.843123 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:15.843094    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c4444dc76-c2dcn_23891ba7-4710-4468-b69d-8764b3cf8384/router/0.log"
Sep 09 20:54:16.871449 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:16.871413    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t552h_dab7b49e-a152-44f0-8ebb-3643d8862b0c/serve-healthcheck-canary/0.log"
Sep 09 20:54:17.487256 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:17.487226    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-gx45b_a32e9cee-8743-486e-a042-41ae301e7f11/insights-operator/0.log"
Sep 09 20:54:17.488029 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:17.488006    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-gx45b_a32e9cee-8743-486e-a042-41ae301e7f11/insights-operator/1.log"
Sep 09 20:54:17.604203 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:17.604169    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mp9c5/perf-node-gather-daemonset-klp5g"
Sep 09 20:54:17.700930 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:17.700903    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sgqkp_aaad6fa0-2376-4fc7-976a-395d763899dc/kube-rbac-proxy/0.log"
Sep 09 20:54:17.731822 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:17.731793    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sgqkp_aaad6fa0-2376-4fc7-976a-395d763899dc/exporter/0.log"
Sep 09 20:54:17.758319 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:17.758240    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sgqkp_aaad6fa0-2376-4fc7-976a-395d763899dc/extractor/0.log"
Sep 09 20:54:20.955128 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:20.955091    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n_3d171978-73e5-4a61-9eaf-8ff6047520fb/manager/0.log"
Sep 09 20:54:21.001584 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:21.001555    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dbdbfdc7-j5c4n_3d171978-73e5-4a61-9eaf-8ff6047520fb/copy-manifests/0.log"
Sep 09 20:54:21.314993 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:21.314960    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758cfc744b-627r6_d14e73dc-c91b-420c-8d6e-d59c9bb38ff0/kube-auth-proxy/0.log"
Sep 09 20:54:21.392010 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:21.391967    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qzrnd_0e70b798-2892-471a-bf91-81797e3ff923/payload-pre-processing/0.log"
Sep 09 20:54:21.423967 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:21.423935    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zdk65_63bec830-c2e4-4a4a-a998-280cbae1baf7/payload-processing/0.log"
Sep 09 20:54:21.439068 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:21.439028    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c4444dc76-c2dcn_23891ba7-4710-4468-b69d-8764b3cf8384/router/0.log"
Sep 09 20:54:22.450447 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:22.450413    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5bc9df88-4gffz_dc84bc43-83d8-4efc-a986-f1835827a75c/manager/0.log"
Sep 09 20:54:22.514321 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:22.514287    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-mvmv6_0345558e-e785-4c86-86d1-63b79652d66e/openshift-lws-operator/0.log"
Sep 09 20:54:23.423708 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:23.423671    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-758cfc744b-627r6_d14e73dc-c91b-420c-8d6e-d59c9bb38ff0/kube-auth-proxy/0.log"
Sep 09 20:54:23.543160 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:23.543120    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qzrnd_0e70b798-2892-471a-bf91-81797e3ff923/payload-pre-processing/0.log"
Sep 09 20:54:23.579131 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:23.579098    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-zdk65_63bec830-c2e4-4a4a-a998-280cbae1baf7/payload-processing/0.log"
Sep 09 20:54:23.602037 ip-10-0-135-122 kubenswrapper[2581]: I0909 20:54:23.602008    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c4444dc76-c2dcn_23891ba7-4710-4468-b69d-8764b3cf8384/router/0.log"