Sep 16 14:35:22.750454 ip-10-0-128-123 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 14:35:23.072373 ip-10-0-128-123 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 14:35:23.072373 ip-10-0-128-123 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 14:35:23.072373 ip-10-0-128-123 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 14:35:23.072373 ip-10-0-128-123 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 14:35:23.072373 ip-10-0-128-123 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 14:35:23.074574 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.074483    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 14:35:23.077818 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077803    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:35:23.077818 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077818    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077823    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077826    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077829    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077832    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077835    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077838    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077841    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077844    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077846    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077849    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077851    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077854    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077857    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077859    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077862    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077865    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077868    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077870    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077874    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:35:23.077882 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077877    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077880    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077882    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077885    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077887    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077890    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077893    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077896    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077899    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077901    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077904    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077907    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077910    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077912    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077915    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077918    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077920    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077923    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077925    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077928    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:35:23.078366 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077930    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077933    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077935    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077937    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077940    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077943    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077946    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077948    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077951    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077953    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077957    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077961    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077963    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077967    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077972    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077975    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077978    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077981    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077984    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:35:23.078848 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077986    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077989    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077992    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077995    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.077998    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078000    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078003    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078006    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078008    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078011    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078014    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078016    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078019    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078022    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078025    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078027    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078030    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078032    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078035    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078038    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:35:23.079319 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078040    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078043    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078045    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078048    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078051    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078053    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078056    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078060    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078062    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078065    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078067    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078070    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078073    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078075    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078078    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078081    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078083    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078088    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078091    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:35:23.079795 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078808    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078815    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078817    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078821    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078824    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078826    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078829    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078832    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078835    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078837    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078840    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078843    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078846    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078848    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078853    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078857    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078861    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078864    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078868    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:35:23.080295 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078872    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078876    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078879    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078882    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078884    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078887    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078890    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078893    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078895    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078898    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078900    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078903    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078907    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078910    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078913    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078915    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078918    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078920    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078923    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078925    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:35:23.080765 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078928    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078930    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078933    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078937    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078939    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078942    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078945    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078948    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078950    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078953    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078955    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078958    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078960    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078963    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078966    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078969    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078972    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078974    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078977    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078980    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:35:23.081267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078982    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078984    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078987    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078990    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078992    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078995    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.078998    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079000    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079003    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079007    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079010    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079013    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079015    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079018    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079020    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079023    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079026    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079028    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079030    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079033    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:35:23.081762 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079036    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079040    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079043    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079046    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079049    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079052    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079055    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079058    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079061    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079063    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079066    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079069    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079072    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079075    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079078    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079081    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079083    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079086    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079089    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:35:23.082277 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079092    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079580    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079586    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079589    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079592    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079595    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079598    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079601    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079604    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079606    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079609    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079613    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079616    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079620    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079623    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079625    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079632    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079634    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079637    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:35:23.082757 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079640    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079642    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079645    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079648    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079650    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079653    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079656    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079658    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079661    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079664    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079666    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079668    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079676    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079679    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079682    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079685    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079687    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079690    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079693    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079695    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:35:23.083223 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079698    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079700    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079702    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079705    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079707    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079710    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079712    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079715    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079717    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079721    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079724    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079726    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079730    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079732    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079735    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079737    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079740    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079742    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079745    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079747    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:35:23.083709 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079750    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079752    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079755    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079757    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079760    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079767    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079770    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079774    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079778    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079780    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079783    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079785    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079788    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079790    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079794    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079796    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079799    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079801    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079804    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:35:23.084199 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079807    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079809    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079814    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079816    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079819    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079821    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079824    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079827    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079829    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079832    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079834    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079837    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079839    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079843    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079846    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079849    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079852    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079855    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079858    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079867    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:35:23.084650 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079870    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.079873    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.079939    2558 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.079973    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.079983    2558 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.079987    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.079991    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.079995    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.079999    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080004    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080007    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080011    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080016    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080020    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080025    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080028    2558 flags.go:64] FLAG: --cgroup-root=""
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080031    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080034    2558 flags.go:64] FLAG: --client-ca-file=""
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080037    2558 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080040    2558 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080046    2558 flags.go:64] FLAG: --cluster-domain=""
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080049    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080052    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080055    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 14:35:23.085177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080058    2558 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080062    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080065    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080068    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080071    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080075    2558 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080078    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080080    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080091    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080094    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080099    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080102    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080105    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080107    2558 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080110    2558 flags.go:64] FLAG: --enable-server="true"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080113    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080120    2558 flags.go:64] FLAG: --event-burst="100"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080123    2558 flags.go:64] FLAG: --event-qps="50"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080126    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080129    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080132    2558 flags.go:64] FLAG: --eviction-hard=""
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080136    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080140    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080143    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080147    2558 flags.go:64] FLAG: --eviction-soft=""
Sep 16 14:35:23.085766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080149    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080152    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080155    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080158    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080173    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080177    2558 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080180    2558 flags.go:64] FLAG: --feature-gates=""
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080184    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080187    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080190    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080193    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080196    2558 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080200    2558 flags.go:64] FLAG: --help="false"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080202    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080206    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080209    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080219    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080223    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080226    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080229    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080232    2558 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080235    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080238    2558 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 14:35:23.086376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080241    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080244    2558 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080247    2558 flags.go:64] FLAG: --kube-reserved=""
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080250    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080253    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080256    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080260    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080265    2558 flags.go:64] FLAG: --lock-file=""
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080268    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080271    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080274    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080279    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080282    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080284    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080287    2558 flags.go:64] FLAG: --logging-format="text"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080290    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080299    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080303    2558 flags.go:64] FLAG: --manifest-url=""
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080306    2558 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080310    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080314    2558 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080318    2558 flags.go:64] FLAG: --max-pods="110"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080321    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080324    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080327    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 14:35:23.086964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080330    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080338    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080342    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080345    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080352    2558 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080355    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080358    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080361    2558 flags.go:64] FLAG: --pod-cidr=""
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080364    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080370    2558 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080373    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080376    2558 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080379    2558 flags.go:64] FLAG: --port="10250"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080382    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080387    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00e4f4982dad4d3db"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080390    2558 flags.go:64] FLAG: --qos-reserved=""
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080394    2558 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080397    2558 flags.go:64] FLAG: --register-node="true"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080399    2558 flags.go:64] FLAG: --register-with-taints=""
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080403    2558 flags.go:64] FLAG: --registry-burst="10"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080407    2558 flags.go:64] FLAG: --registry-qps="5"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080410    2558 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080412    2558 flags.go:64] FLAG: --reserved-memory=""
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080416    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080420    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 14:35:23.087833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080423    2558 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080425    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080428    2558 flags.go:64] FLAG: --runonce="false"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080431    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080434    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080437    2558 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080440    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080443    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080446    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080455    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080458    2558 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080461    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080464    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080467    2558 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080469    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080472    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080476    2558 flags.go:64] FLAG: --system-cgroups=""
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080479    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080484    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080487    2558 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080490    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080502    2558 flags.go:64] FLAG: --tls-min-version=""
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080505    2558 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080508    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080511    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 14:35:23.088711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080514    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080517    2558 flags.go:64] FLAG: --v="2"
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080521    2558 flags.go:64] FLAG: --version="false"
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080525    2558 flags.go:64] FLAG: --vmodule=""
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080529    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.080532    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080690    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080694    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080697    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080700    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080703    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080706    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080710    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080712    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080715    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080718    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080720    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080724    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080727    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080729    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080732    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:35:23.089368 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080735    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080737    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080740    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080742    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080745    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080747    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080750    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080755    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080758    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080761    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080763    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080766    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080769    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080772    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080774    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080777    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080780    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080782    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080785    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080788    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:35:23.089888 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080790    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080793    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080796    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080798    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080801    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080804    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080806    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080809    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080811    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080814    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080817    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080819    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080822    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080825    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080827    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080830    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080834    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080837    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080840    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080846    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:35:23.090401 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080849    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080852    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080854    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080857    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080860    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080863    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080865    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080868    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080870    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080874    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080877    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080880    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080883    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080886    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080889    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080892    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080895    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080898    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080901    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:35:23.090902 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080904    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080906    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080909    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080912    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080915    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080918    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080920    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080923    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080925    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080928    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080931    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080933    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080938    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080941    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080944    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080946    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080949    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080952    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080954    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080957    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:35:23.091378 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080959    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080962    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080965    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080967    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.080970    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.081426    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.089208    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.089225    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.089713    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.089720    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089778    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089783    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089787    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089791    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089794    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:35:23.091866 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089797    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089800    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089803    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089805    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089808    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089810    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089813    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089816    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089819    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089821    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089824    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089827    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089830    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089833    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089836    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089840    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089844    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089846    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089849    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:35:23.092248 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089851    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089854    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089857    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089860    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089864    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089868    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089870    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089875    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089879    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089882    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089885    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089888    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089891    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089894    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089897    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089900    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089903    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089906    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089909    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089912    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:35:23.092756 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089914    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089917    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089920    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089923    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089927    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089930    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089932    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089935    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089938    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089941    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089943    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089946    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089948    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089951    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089954    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089957    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089959    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089962    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089965    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089968    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:35:23.093350 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089970    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089973    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089975    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089978    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089981    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089983    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089986    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089988    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089991    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089994    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089996    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.089999    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090002    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090004    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090007    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090010    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090013    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090016    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090019    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:35:23.093895 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090021    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090024    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090027    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090030    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090032    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090034    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090037    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090040    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090042    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090045    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090048    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090050    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090054    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090057    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090060    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:35:23.094477 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090063    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.090068    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090201    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090207    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090211    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090214    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090217    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090220    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090223    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090226    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090229    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090233    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090236    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090239    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090243    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 14:35:23.094918 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090245    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090250    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090253    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090256    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090260    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090265    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090268    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090271    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090274    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090276    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090279    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090282    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090285    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090288    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090290    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090293    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090295    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090298    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090301    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 14:35:23.095318 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090303    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090306    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090309    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090312    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090314    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090317    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090319    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090322    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090325    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090327    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090330    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090332    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090335    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090338    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090340    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090343    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090346    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090348    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090351    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090353    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 14:35:23.096047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090356    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090359    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090361    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090364    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090366    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090369    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090371    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090374    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090376    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090379    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090381    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090384    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090386    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090389    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090391    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090394    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090397    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090399    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090402    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 14:35:23.096737 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090404    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090407    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090410    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090413    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090415    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090418    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090421    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090424    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090427    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090429    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090432    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090434    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090437    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090440    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090442    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090445    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090447    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090450    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090452    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090455    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 14:35:23.097276 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090458    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090460    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090463    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090465    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090468    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090470    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090473    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:23.090475    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.090480    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.090591    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.093239    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.093971    2558 server.go:1019] "Starting client certificate rotation"
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.094083    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.094133    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 14:35:23.097835 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.096104    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 14:35:23.098987 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.098967    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 14:35:23.106285 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.106268    2558 log.go:25] "Validated CRI v1 runtime API"
Sep 16 14:35:23.108887 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.108872    2558 log.go:25] "Validated CRI v1 image API"
Sep 16 14:35:23.109666 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.109653    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 14:35:23.112697 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.112676    2558 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ac60d983-7203-43ff-b041-5e11a231333f:/dev/nvme0n1p4 c8f2cd15-083b-48d5-a0e0-bfdee76cda16:/dev/nvme0n1p3]
Sep 16 14:35:23.112758 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.112697    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 14:35:23.119507 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.119395    2558 manager.go:216] Machine: {Timestamp:2026-09-16 14:35:23.118399792 +0000 UTC m=+0.272478070 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105267 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a67aaf48d6d4ad99a3881e8441cd8 SystemUUID:ec2a67aa-f48d-6d4a-d99a-3881e8441cd8 BootID:e1469ca6-153e-474e-80e6-3a8acd7d5766 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:26:5f:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:26:5f:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:88:f7:b9:ae:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 14:35:23.119507 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.119499    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 14:35:23.119650 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.119586    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 14:35:23.120388 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.120368    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 14:35:23.120515 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.120391    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 14:35:23.120946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.120936    2558 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 14:35:23.120979 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.120948    2558 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 14:35:23.120979 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.120956    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 14:35:23.120979 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.120970    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 14:35:23.121634 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.121625    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 14:35:23.125265 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.125244    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 14:35:23.125381 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.125372    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 14:35:23.125429 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.125409    2558 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 14:35:23.125429 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.125419    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 14:35:23.125496 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.125434    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 14:35:23.125496 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.125444    2558 kubelet.go:398] "Adding apiserver pod source"
Sep 16 14:35:23.125496 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.125454    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 14:35:23.125844 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.125827    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 14:35:23.126329 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.126316    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 14:35:23.126408 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.126335    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 14:35:23.127977 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.127960    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 14:35:23.129974 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.129959    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 14:35:23.130041 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.129990    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 14:35:23.131321 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131309    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 14:35:23.131373 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131324    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 14:35:23.131373 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131330    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 14:35:23.131373 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131336    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 14:35:23.131373 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131345    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 14:35:23.131373 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131352    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 14:35:23.131373 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131368    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 14:35:23.131580 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131377    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 14:35:23.131580 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131385    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 14:35:23.131580 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131390    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 14:35:23.131580 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131403    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 14:35:23.131580 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.131409    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 14:35:23.132052 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.132038    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 14:35:23.132052 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.132049    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 14:35:23.135315 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.135262    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 14:35:23.135407 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.135316    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 14:35:23.135407 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.135325    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-123.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 14:35:23.135864 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.135852    2558 server.go:1268] "Started kubelet"
Sep 16 14:35:23.135917 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.135892    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 14:35:23.136232 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.136047    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 14:35:23.136482 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.136467    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 14:35:23.136927 ip-10-0-128-123 systemd[1]: Started Kubernetes Kubelet.
Sep 16 14:35:23.137678 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.137649    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 14:35:23.138552 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.138538    2558 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 14:35:23.139970 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.139948    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 14:35:23.143546 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.143532    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 14:35:23.143628 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.143550    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 14:35:23.144107 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.143670    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-123.ec2.internal\" not found"
Sep 16 14:35:23.144107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.144075    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 14:35:23.144302 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.144115    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 14:35:23.144302 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.144127    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 14:35:23.144398 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.144368    2558 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 14:35:23.144398 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.144377    2558 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 14:35:23.144951 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.144779    2558 factory.go:153] Registering CRI-O factory
Sep 16 14:35:23.145038 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.144954    2558 factory.go:223] Registration of the crio container factory successfully
Sep 16 14:35:23.145038 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.144994    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 14:35:23.145038 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.145001    2558 factory.go:55] Registering systemd factory
Sep 16 14:35:23.145038 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.145007    2558 factory.go:223] Registration of the systemd container factory successfully
Sep 16 14:35:23.145038 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.145019    2558 factory.go:103] Registering Raw factory
Sep 16 14:35:23.145038 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.145029    2558 manager.go:1196] Started watching for new ooms in manager
Sep 16 14:35:23.145848 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.145826    2558 manager.go:349] Starting recovery of all containers
Sep 16 14:35:23.148046 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.148012    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 14:35:23.153223 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.153187    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 14:35:23.153614 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.153597    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f8qrq"
Sep 16 14:35:23.154214 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.153235    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-123.ec2.internal.18d5d3d15e727d40  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-123.ec2.internal,UID:ip-10-0-128-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-123.ec2.internal,},FirstTimestamp:2026-09-16 14:35:23.135827264 +0000 UTC m=+0.289905609,LastTimestamp:2026-09-16 14:35:23.135827264 +0000 UTC m=+0.289905609,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-123.ec2.internal,}"
Sep 16 14:35:23.155339 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.155297    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 14:35:23.158359 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.158322    2558 manager.go:354] Recovery completed
Sep 16 14:35:23.163527 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.163514    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:35:23.165996 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.165980    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f8qrq"
Sep 16 14:35:23.166310 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.166296    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:35:23.166366 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.166325    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:35:23.166366 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.166340    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:35:23.168049 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.167995    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 14:35:23.168049 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.168007    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 14:35:23.168049 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.168024    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 14:35:23.168306 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.168291    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:35:23.168649 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.168523    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-123.ec2.internal.18d5d3d16043a28b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-123.ec2.internal,UID:ip-10-0-128-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-123.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-123.ec2.internal,},FirstTimestamp:2026-09-16 14:35:23.166311051 +0000 UTC m=+0.320389329,LastTimestamp:2026-09-16 14:35:23.166311051 +0000 UTC m=+0.320389329,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-123.ec2.internal,}"
Sep 16 14:35:23.169246 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.169231    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:35:23.169309 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.169259    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:35:23.169309 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.169272    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:35:23.170331 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.170311    2558 policy_none.go:49] "None policy: Start"
Sep 16 14:35:23.170331 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.170327    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 14:35:23.170433 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.170337    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 14:35:23.172364 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.172350    2558 policy_none.go:47] "Start"
Sep 16 14:35:23.200896 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.200880    2558 manager.go:337] "Starting Device Plugin manager"
Sep 16 14:35:23.226255 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.200916    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 14:35:23.226255 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.200925    2558 server.go:87] "Starting device plugin registration server"
Sep 16 14:35:23.226255 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.201303    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 14:35:23.226255 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.201314    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 14:35:23.226255 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.201425    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 14:35:23.226255 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.201504    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 14:35:23.226255 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.201511    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 14:35:23.226255 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.202007    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 14:35:23.226255 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.202048    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-123.ec2.internal\" not found"
Sep 16 14:35:23.288531 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.288507    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 14:35:23.288638 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.288569    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 14:35:23.288638 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.288596    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 14:35:23.288804 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.288643    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 14:35:23.290963 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.290942    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:35:23.301966 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.301950    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:35:23.302791 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.302776    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:35:23.302870 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.302806    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:35:23.302870 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.302820    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:35:23.302870 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.302841    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.311192 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.311177    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.311267 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.311199    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-123.ec2.internal\": node \"ip-10-0-128-123.ec2.internal\" not found"
Sep 16 14:35:23.328352 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.326940    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-123.ec2.internal\" not found"
Sep 16 14:35:23.389520 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.389463    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-123.ec2.internal"]
Sep 16 14:35:23.389689 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.389572    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:35:23.390606 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.390582    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:35:23.390703 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.390615    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:35:23.390703 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.390627    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:35:23.393069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.393056    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:35:23.393197 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.393183    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.393252 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.393222    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:35:23.393790 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.393773    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:35:23.393901 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.393795    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:35:23.393901 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.393806    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:35:23.393901 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.393846    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:35:23.393901 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.393872    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:35:23.393901 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.393886    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:35:23.396137 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.396119    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.396244 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.396148    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 14:35:23.396841 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.396825    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 14:35:23.396906 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.396853    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 14:35:23.396906 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.396895    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeHasSufficientPID"
Sep 16 14:35:23.420845 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.420821    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-123.ec2.internal\" not found" node="ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.425214 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.425197    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-123.ec2.internal\" not found" node="ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.427278 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.427263    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-123.ec2.internal\" not found"
Sep 16 14:35:23.446034 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.446014    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd5590f8cd50c811787920daad48d22d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal\" (UID: \"cd5590f8cd50c811787920daad48d22d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.446142 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.446042    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd5590f8cd50c811787920daad48d22d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal\" (UID: \"cd5590f8cd50c811787920daad48d22d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.446142 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.446066    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e99847c9ed9bfebb0b4989bf767e4cc0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-123.ec2.internal\" (UID: \"e99847c9ed9bfebb0b4989bf767e4cc0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.527677 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.527645    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-123.ec2.internal\" not found"
Sep 16 14:35:23.546194 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.546150    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd5590f8cd50c811787920daad48d22d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal\" (UID: \"cd5590f8cd50c811787920daad48d22d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.546330 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.546200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd5590f8cd50c811787920daad48d22d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal\" (UID: \"cd5590f8cd50c811787920daad48d22d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.546330 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.546255    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd5590f8cd50c811787920daad48d22d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal\" (UID: \"cd5590f8cd50c811787920daad48d22d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.546330 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.546316    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cd5590f8cd50c811787920daad48d22d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal\" (UID: \"cd5590f8cd50c811787920daad48d22d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.546447 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.546354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e99847c9ed9bfebb0b4989bf767e4cc0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-123.ec2.internal\" (UID: \"e99847c9ed9bfebb0b4989bf767e4cc0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.546447 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.546385    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e99847c9ed9bfebb0b4989bf767e4cc0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-123.ec2.internal\" (UID: \"e99847c9ed9bfebb0b4989bf767e4cc0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.628680 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.628610    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-123.ec2.internal\" not found"
Sep 16 14:35:23.723110 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.723082    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.728055 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.727950    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:23.729013 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.728999    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-123.ec2.internal\" not found"
Sep 16 14:35:23.830090 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.830050    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-123.ec2.internal\" not found"
Sep 16 14:35:23.930618 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:23.930539    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-123.ec2.internal\" not found"
Sep 16 14:35:23.983615 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:23.983584    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:35:24.032627 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.032590    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-123.ec2.internal\" not found"
Sep 16 14:35:24.063124 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.063092    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:35:24.093806 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.093762    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 14:35:24.094338 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.093957    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 14:35:24.094338 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.093986    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 14:35:24.094338 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.093986    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 14:35:24.127681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.127640    2558 apiserver.go:52] "Watching apiserver"
Sep 16 14:35:24.144345 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.144322    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 14:35:24.144345 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.144344    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:24.146363 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.146342    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 14:35:24.146738 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.146716    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-j8tbw","openshift-dns/node-resolver-jzrt4","openshift-image-registry/node-ca-vpphh","openshift-multus/multus-fzpp9","openshift-network-diagnostics/network-check-target-n4fxc","kube-system/global-pull-secret-syncer-r9l6v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t","openshift-cluster-node-tuning-operator/tuned-4wzzx","openshift-multus/multus-additional-cni-plugins-6pmdn","openshift-multus/network-metrics-daemon-qwq9w","openshift-network-operator/iptables-alerter-wt2qd","openshift-ovn-kubernetes/ovnkube-node-vskvj"]
Sep 16 14:35:24.149446 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.149424    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:24.151704 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.151684    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jzrt4"
Sep 16 14:35:24.152406 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.152391    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 14:35:24.152697 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.152681    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qzb5x\""
Sep 16 14:35:24.152788 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.152699    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 14:35:24.153989 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.153972    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vpphh"
Sep 16 14:35:24.154434 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.154416    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 14:35:24.154548 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.154453    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 14:35:24.154548 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.154467    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nq28b\""
Sep 16 14:35:24.156321 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.156308    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.158396 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.158378    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 14:35:24.158507 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.158388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lzrlt\""
Sep 16 14:35:24.158507 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.158396    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 14:35:24.159203 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.158846    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 14:35:24.159203 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.159003    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-123.ec2.internal"
Sep 16 14:35:24.159203 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.159126    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:24.159429 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.159230    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:24.159701 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.159671    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8jjlz\""
Sep 16 14:35:24.159803 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.159712    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 14:35:24.159803 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.159743    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 14:35:24.160038 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.160020    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 14:35:24.160108 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.160084    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 14:35:24.160173 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.160127    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 14:35:24.161749 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.161727    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 14:35:24.161952 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.161935    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:24.162009 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.161994    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:24.164385 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.164362    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.166981 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.166966    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.168420 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.168401    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 14:35:24.168847 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.168831    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 14:35:24.168935 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.168925    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-94nwr\""
Sep 16 14:35:24.169034 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.169019    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 14:35:24.169564 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.169548    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.169939 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.169924    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 14:35:24.169939 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.169930    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-swlm9\""
Sep 16 14:35:24.169939 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.169937    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:35:24.171023 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.170986    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 14:30:23 +0000 UTC" deadline="2028-02-28 01:56:02.024322035 +0000 UTC"
Sep 16 14:35:24.171076 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.171024    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12707h20m37.853300992s"
Sep 16 14:35:24.171851 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.171836    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:24.171905 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.171890    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:24.173016 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.173000    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 14:35:24.173113 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.173042    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 14:35:24.173173 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.173123    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 14:35:24.173173 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.173137    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wfwkk\""
Sep 16 14:35:24.174218 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.174204    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wt2qd"
Sep 16 14:35:24.176647 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.176628    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal"]
Sep 16 14:35:24.176708 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.176655    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-123.ec2.internal"]
Sep 16 14:35:24.176766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.176751    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.178432 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.178414    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 14:35:24.178529 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.178422    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 14:35:24.178951 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.178934    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xpn8q\""
Sep 16 14:35:24.179026 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.178970    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:35:24.179436 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.179416    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vz2m7\""
Sep 16 14:35:24.179535 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.179472    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 14:35:24.179783 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.179763    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 14:35:24.179876 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.179814    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 14:35:24.179876 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.179817    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 14:35:24.181075 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.181033    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 14:35:24.181145 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.181120    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 14:35:24.184150 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.184134    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l5xws"
Sep 16 14:35:24.196921 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.196901    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l5xws"
Sep 16 14:35:24.245675 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.245654    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 14:35:24.249573 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249553    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-run-multus-certs\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.249626 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249582    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-tuned\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.249626 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249600    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be55def-d862-4674-864b-04119c40ceeb-host\") pod \"node-ca-vpphh\" (UID: \"9be55def-d862-4674-864b-04119c40ceeb\") " pod="openshift-image-registry/node-ca-vpphh"
Sep 16 14:35:24.249720 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249623    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9be55def-d862-4674-864b-04119c40ceeb-serviceca\") pod \"node-ca-vpphh\" (UID: \"9be55def-d862-4674-864b-04119c40ceeb\") " pod="openshift-image-registry/node-ca-vpphh"
Sep 16 14:35:24.249720 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249654    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b24lj\" (UniqueName: \"kubernetes.io/projected/b80910f2-ec56-4626-bf5e-7db8a49f9765-kube-api-access-b24lj\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.249720 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249694    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.249819 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249745    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.249819 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-modprobe-d\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.249819 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249810    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-conf-dir\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.249946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249831    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-kubelet\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.249946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249847    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-run-ovn\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.249946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249862    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-run-ovn-kubernetes\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.249946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249907    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-cnibin\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.249946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249942    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-sysctl-d\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.250198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249967    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-systemd\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.250198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.249993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e643bf43-b88e-4f36-ae22-f0957a4864b2-tmp\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.250198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-cni-bin\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.250198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250045    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-etc-selinux\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.250198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250074    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-etc-openvswitch\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.250198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250098    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-log-socket\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.250198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:24.250198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250153    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96965f62-4546-4c71-89e6-d3f78095e3f9-iptables-alerter-script\") pod \"iptables-alerter-wt2qd\" (UID: \"96965f62-4546-4c71-89e6-d3f78095e3f9\") " pod="openshift-network-operator/iptables-alerter-wt2qd"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250243    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-socket-dir-parent\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-kubernetes\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250290    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-host\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250315    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82cdd79b-96ab-48dd-a670-0d59194a7992-kubelet-config\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250341    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0fa62dc0-5f61-4349-b7c6-9d4242503566-hosts-file\") pod \"node-resolver-jzrt4\" (UID: \"0fa62dc0-5f61-4349-b7c6-9d4242503566\") " pod="openshift-dns/node-resolver-jzrt4"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250366    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-var-lib-kubelet\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250388    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-device-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250414    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-var-lib-cni-bin\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250442    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-var-lib-kubelet\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250457    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-node-log\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250513    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-run\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250528    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7qv\" (UniqueName: \"kubernetes.io/projected/e643bf43-b88e-4f36-ae22-f0957a4864b2-kube-api-access-zp7qv\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.250550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250543    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-run-netns\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250565    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1641169-72fe-4cfb-b569-60d355a4f551-agent-certs\") pod \"konnectivity-agent-j8tbw\" (UID: \"e1641169-72fe-4cfb-b569-60d355a4f551\") " pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250584    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1641169-72fe-4cfb-b569-60d355a4f551-konnectivity-ca\") pod \"konnectivity-agent-j8tbw\" (UID: \"e1641169-72fe-4cfb-b569-60d355a4f551\") " pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250600    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8nnj\" (UniqueName: \"kubernetes.io/projected/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-kube-api-access-m8nnj\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250615    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-hostroot\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250634    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-env-overrides\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250664    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t99ct\" (UniqueName: \"kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct\") pod \"network-check-target-n4fxc\" (UID: \"ecbbbd3b-6571-414d-94c1-130a385f4d1a\") " pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250689    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-socket-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-cni-binary-copy\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250747    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-system-cni-dir\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250786    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e679c20-c581-4d8c-9487-e8b2ae016ec4-cni-binary-copy\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250817    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-var-lib-cni-multus\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250842    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-lib-modules\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250864    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-registration-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250897    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-ovn-node-metrics-cert\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250932    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-daemon-config\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.251149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250957    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-sysconfig\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.251924 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.250979    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-systemd-units\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.251924 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.251003    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-sys-fs\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.251924 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.251033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-cnibin\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.251924 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.251055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh8st\" (UniqueName: \"kubernetes.io/projected/b52afd57-0df9-4a81-b6dc-a90212fcc659-kube-api-access-wh8st\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:24.251924 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.251071    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96965f62-4546-4c71-89e6-d3f78095e3f9-host-slash\") pod \"iptables-alerter-wt2qd\" (UID: \"96965f62-4546-4c71-89e6-d3f78095e3f9\") " pod="openshift-network-operator/iptables-alerter-wt2qd"
Sep 16 14:35:24.251924 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.251095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-run-netns\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.251924 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.251117    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fa62dc0-5f61-4349-b7c6-9d4242503566-tmp-dir\") pod \"node-resolver-jzrt4\" (UID: \"0fa62dc0-5f61-4349-b7c6-9d4242503566\") " pod="openshift-dns/node-resolver-jzrt4"
Sep 16 14:35:24.251924 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.251140    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b7lr\" (UniqueName: \"kubernetes.io/projected/0fa62dc0-5f61-4349-b7c6-9d4242503566-kube-api-access-6b7lr\") pod \"node-resolver-jzrt4\" (UID: \"0fa62dc0-5f61-4349-b7c6-9d4242503566\") " pod="openshift-dns/node-resolver-jzrt4"
Sep 16 14:35:24.251924 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.251156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-sysctl-conf\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.251924 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.251196    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-ovnkube-config\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.251924 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.251687    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.251755    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-run-k8s-cni-cncf-io\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.252757    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-var-lib-openvswitch\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.252800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-ovnkube-script-lib\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.252831    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82cdd79b-96ab-48dd-a670-0d59194a7992-dbus\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.252862    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99w6l\" (UniqueName: \"kubernetes.io/projected/4e679c20-c581-4d8c-9487-e8b2ae016ec4-kube-api-access-99w6l\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.252887    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-sys\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.252918    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-slash\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.252946    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-run-systemd\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.252974    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-run-openvswitch\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.253004    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gnsr\" (UniqueName: \"kubernetes.io/projected/9be55def-d862-4674-864b-04119c40ceeb-kube-api-access-5gnsr\") pod \"node-ca-vpphh\" (UID: \"9be55def-d862-4674-864b-04119c40ceeb\") " pod="openshift-image-registry/node-ca-vpphh"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.253032    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjm4\" (UniqueName: \"kubernetes.io/projected/96965f62-4546-4c71-89e6-d3f78095e3f9-kube-api-access-5pjm4\") pod \"iptables-alerter-wt2qd\" (UID: \"96965f62-4546-4c71-89e6-d3f78095e3f9\") " pod="openshift-network-operator/iptables-alerter-wt2qd"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.253061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-cni-dir\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.253089    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6vq\" (UniqueName: \"kubernetes.io/projected/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-kube-api-access-xw6vq\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.253120    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-system-cni-dir\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.253148    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-etc-kubernetes\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.254189 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.253172    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-cni-netd\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.254795 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.253217    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.344345 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.344319    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:35:24.353788 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.353764    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-var-lib-cni-bin\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.353899 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.353798    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-var-lib-kubelet\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.353899 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.353814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-node-log\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.353899 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.353833    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.353899 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.353886    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-var-lib-cni-bin\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.354069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.353908    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-var-lib-kubelet\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.354069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.353923    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-run\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.354069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.353949    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.354069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.353952    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7qv\" (UniqueName: \"kubernetes.io/projected/e643bf43-b88e-4f36-ae22-f0957a4864b2-kube-api-access-zp7qv\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.354069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.353979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-run-netns\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.354069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1641169-72fe-4cfb-b569-60d355a4f551-agent-certs\") pod \"konnectivity-agent-j8tbw\" (UID: \"e1641169-72fe-4cfb-b569-60d355a4f551\") " pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:24.354069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-node-log\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.354069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.353977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-run\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.354069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354026    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1641169-72fe-4cfb-b569-60d355a4f551-konnectivity-ca\") pod \"konnectivity-agent-j8tbw\" (UID: \"e1641169-72fe-4cfb-b569-60d355a4f551\") " pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:24.354069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354032    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-run-netns\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354191    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8nnj\" (UniqueName: \"kubernetes.io/projected/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-kube-api-access-m8nnj\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354231    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-hostroot\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354256    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-env-overrides\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354281    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99ct\" (UniqueName: \"kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct\") pod \"network-check-target-n4fxc\" (UID: \"ecbbbd3b-6571-414d-94c1-130a385f4d1a\") " pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354295    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-hostroot\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-socket-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-cni-binary-copy\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-socket-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354477    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-system-cni-dir\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e679c20-c581-4d8c-9487-e8b2ae016ec4-cni-binary-copy\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354537    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-var-lib-cni-multus\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.354553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-lib-modules\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354583    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-registration-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354609    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-ovn-node-metrics-cert\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354605    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354632    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-daemon-config\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354620    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-var-lib-cni-multus\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354663    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-sysconfig\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-systemd-units\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-sys-fs\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354759    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-cnibin\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-lib-modules\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354778    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-system-cni-dir\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354790    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh8st\" (UniqueName: \"kubernetes.io/projected/b52afd57-0df9-4a81-b6dc-a90212fcc659-kube-api-access-wh8st\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354821    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96965f62-4546-4c71-89e6-d3f78095e3f9-host-slash\") pod \"iptables-alerter-wt2qd\" (UID: \"96965f62-4546-4c71-89e6-d3f78095e3f9\") " pod="openshift-network-operator/iptables-alerter-wt2qd"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354828    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-registration-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354845    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-sysconfig\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354848    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1641169-72fe-4cfb-b569-60d355a4f551-konnectivity-ca\") pod \"konnectivity-agent-j8tbw\" (UID: \"e1641169-72fe-4cfb-b569-60d355a4f551\") " pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-run-netns\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.355107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fa62dc0-5f61-4349-b7c6-9d4242503566-tmp-dir\") pod \"node-resolver-jzrt4\" (UID: \"0fa62dc0-5f61-4349-b7c6-9d4242503566\") " pod="openshift-dns/node-resolver-jzrt4"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354901    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-cnibin\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354922    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7lr\" (UniqueName: \"kubernetes.io/projected/0fa62dc0-5f61-4349-b7c6-9d4242503566-kube-api-access-6b7lr\") pod \"node-resolver-jzrt4\" (UID: \"0fa62dc0-5f61-4349-b7c6-9d4242503566\") " pod="openshift-dns/node-resolver-jzrt4"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354957    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-sysctl-conf\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354986    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-ovnkube-config\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355006    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-run-netns\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355031    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355034    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96965f62-4546-4c71-89e6-d3f78095e3f9-host-slash\") pod \"iptables-alerter-wt2qd\" (UID: \"96965f62-4546-4c71-89e6-d3f78095e3f9\") " pod="openshift-network-operator/iptables-alerter-wt2qd"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354957    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-systemd-units\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355059    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-run-k8s-cni-cncf-io\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355103    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-var-lib-openvswitch\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355149    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-ovnkube-script-lib\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4e679c20-c581-4d8c-9487-e8b2ae016ec4-cni-binary-copy\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355217    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82cdd79b-96ab-48dd-a670-0d59194a7992-dbus\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355246    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99w6l\" (UniqueName: \"kubernetes.io/projected/4e679c20-c581-4d8c-9487-e8b2ae016ec4-kube-api-access-99w6l\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355292    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-sys\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.355300    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355315    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fa62dc0-5f61-4349-b7c6-9d4242503566-tmp-dir\") pod \"node-resolver-jzrt4\" (UID: \"0fa62dc0-5f61-4349-b7c6-9d4242503566\") " pod="openshift-dns/node-resolver-jzrt4"
Sep 16 14:35:24.355910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355318    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-sysctl-conf\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355319    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-slash\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355355    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-daemon-config\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355104    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-sys-fs\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.354992    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-cni-binary-copy\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355382    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-slash\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.355404    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs podName:b52afd57-0df9-4a81-b6dc-a90212fcc659 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:24.85535903 +0000 UTC m=+2.009437316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs") pod "network-metrics-daemon-qwq9w" (UID: "b52afd57-0df9-4a81-b6dc-a90212fcc659") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/82cdd79b-96ab-48dd-a670-0d59194a7992-dbus\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355451    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-run-systemd\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355490    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-var-lib-openvswitch\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355484    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-run-openvswitch\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355540    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gnsr\" (UniqueName: \"kubernetes.io/projected/9be55def-d862-4674-864b-04119c40ceeb-kube-api-access-5gnsr\") pod \"node-ca-vpphh\" (UID: \"9be55def-d862-4674-864b-04119c40ceeb\") " pod="openshift-image-registry/node-ca-vpphh"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355567    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjm4\" (UniqueName: \"kubernetes.io/projected/96965f62-4546-4c71-89e6-d3f78095e3f9-kube-api-access-5pjm4\") pod \"iptables-alerter-wt2qd\" (UID: \"96965f62-4546-4c71-89e6-d3f78095e3f9\") " pod="openshift-network-operator/iptables-alerter-wt2qd"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355571    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-run-systemd\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355593    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-run-openvswitch\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-cni-dir\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355595    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-cni-dir\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.356769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6vq\" (UniqueName: \"kubernetes.io/projected/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-kube-api-access-xw6vq\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355732    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-system-cni-dir\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355810    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-etc-kubernetes\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355838    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-cni-netd\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355892    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-run-multus-certs\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355918    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-tuned\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355944    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be55def-d862-4674-864b-04119c40ceeb-host\") pod \"node-ca-vpphh\" (UID: \"9be55def-d862-4674-864b-04119c40ceeb\") " pod="openshift-image-registry/node-ca-vpphh"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355971    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9be55def-d862-4674-864b-04119c40ceeb-serviceca\") pod \"node-ca-vpphh\" (UID: \"9be55def-d862-4674-864b-04119c40ceeb\") " pod="openshift-image-registry/node-ca-vpphh"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355991    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-etc-kubernetes\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b24lj\" (UniqueName: \"kubernetes.io/projected/b80910f2-ec56-4626-bf5e-7db8a49f9765-kube-api-access-b24lj\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356028    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356051    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-system-cni-dir\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356081    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356106    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be55def-d862-4674-864b-04119c40ceeb-host\") pod \"node-ca-vpphh\" (UID: \"9be55def-d862-4674-864b-04119c40ceeb\") " pod="openshift-image-registry/node-ca-vpphh"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356110    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-modprobe-d\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356169    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-sys\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.357731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356208    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-cni-netd\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-conf-dir\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356256    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-ovnkube-config\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356289    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-kubelet\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356313    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-modprobe-d\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356315    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-run-ovn\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356362    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-run-ovn-kubernetes\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356393    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-cnibin\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356420    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-sysctl-d\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356459    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-systemd\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356317    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356497    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e643bf43-b88e-4f36-ae22-f0957a4864b2-tmp\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.355969    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-ovnkube-script-lib\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356561    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9be55def-d862-4674-864b-04119c40ceeb-serviceca\") pod \"node-ca-vpphh\" (UID: \"9be55def-d862-4674-864b-04119c40ceeb\") " pod="openshift-image-registry/node-ca-vpphh"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356583    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-run-ovn-kubernetes\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356597    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-conf-dir\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356565    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-cni-bin\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356626    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-cnibin\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.358621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356648    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-systemd\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-etc-selinux\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-run-ovn\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356669    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-cni-bin\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356694    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-etc-openvswitch\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356724    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-log-socket\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356751    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-sysctl-d\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356778    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356806    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96965f62-4546-4c71-89e6-d3f78095e3f9-iptables-alerter-script\") pod \"iptables-alerter-wt2qd\" (UID: \"96965f62-4546-4c71-89e6-d3f78095e3f9\") " pod="openshift-network-operator/iptables-alerter-wt2qd"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356812    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-host-kubelet\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-socket-dir-parent\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356869    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-etc-selinux\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356929    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-run-k8s-cni-cncf-io\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.356980    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357008    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-multus-socket-dir-parent\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.359327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.357062    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret podName:82cdd79b-96ab-48dd-a670-0d59194a7992 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:24.857043933 +0000 UTC m=+2.011122206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret") pod "global-pull-secret-syncer-r9l6v" (UID: "82cdd79b-96ab-48dd-a670-0d59194a7992") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356874    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-kubernetes\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357089    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357109    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-host\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357106    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-kubernetes\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.356315    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-run-multus-certs\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357137    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82cdd79b-96ab-48dd-a670-0d59194a7992-kubelet-config\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357139    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-etc-openvswitch\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357164    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0fa62dc0-5f61-4349-b7c6-9d4242503566-hosts-file\") pod \"node-resolver-jzrt4\" (UID: \"0fa62dc0-5f61-4349-b7c6-9d4242503566\") " pod="openshift-dns/node-resolver-jzrt4"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357175    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e643bf43-b88e-4f36-ae22-f0957a4864b2-host\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357191    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-var-lib-kubelet\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357196    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/82cdd79b-96ab-48dd-a670-0d59194a7992-kubelet-config\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357221    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-device-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357251    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-log-socket\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357269    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0fa62dc0-5f61-4349-b7c6-9d4242503566-hosts-file\") pod \"node-resolver-jzrt4\" (UID: \"0fa62dc0-5f61-4349-b7c6-9d4242503566\") " pod="openshift-dns/node-resolver-jzrt4"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357424    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b80910f2-ec56-4626-bf5e-7db8a49f9765-device-dir\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.360083 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357484    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-env-overrides\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.360922 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e679c20-c581-4d8c-9487-e8b2ae016ec4-host-var-lib-kubelet\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.360922 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.357639    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96965f62-4546-4c71-89e6-d3f78095e3f9-iptables-alerter-script\") pod \"iptables-alerter-wt2qd\" (UID: \"96965f62-4546-4c71-89e6-d3f78095e3f9\") " pod="openshift-network-operator/iptables-alerter-wt2qd"
Sep 16 14:35:24.360922 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.358965    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e643bf43-b88e-4f36-ae22-f0957a4864b2-tmp\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.360922 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.359213    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-ovn-node-metrics-cert\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.360922 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.359331    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1641169-72fe-4cfb-b569-60d355a4f551-agent-certs\") pod \"konnectivity-agent-j8tbw\" (UID: \"e1641169-72fe-4cfb-b569-60d355a4f551\") " pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:24.360922 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.359360    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e643bf43-b88e-4f36-ae22-f0957a4864b2-etc-tuned\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.365831 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.365812    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:35:24.365933 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.365835    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:35:24.365933 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.365848    2558 projected.go:196] Error preparing data for projected volume kube-api-access-t99ct for pod openshift-network-diagnostics/network-check-target-n4fxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:24.365933 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.365905    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct podName:ecbbbd3b-6571-414d-94c1-130a385f4d1a nodeName:}" failed. No retries permitted until 2026-09-16 14:35:24.865888466 +0000 UTC m=+2.019966733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t99ct" (UniqueName: "kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct") pod "network-check-target-n4fxc" (UID: "ecbbbd3b-6571-414d-94c1-130a385f4d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:24.367052 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.367032    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7qv\" (UniqueName: \"kubernetes.io/projected/e643bf43-b88e-4f36-ae22-f0957a4864b2-kube-api-access-zp7qv\") pod \"tuned-4wzzx\" (UID: \"e643bf43-b88e-4f36-ae22-f0957a4864b2\") " pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.367742 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.367691    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh8st\" (UniqueName: \"kubernetes.io/projected/b52afd57-0df9-4a81-b6dc-a90212fcc659-kube-api-access-wh8st\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:24.367742 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.367729    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8nnj\" (UniqueName: \"kubernetes.io/projected/b9e4726b-a5cf-4ae6-93ff-bb145d6e1340-kube-api-access-m8nnj\") pod \"multus-additional-cni-plugins-6pmdn\" (UID: \"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340\") " pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.371287 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.371265    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b7lr\" (UniqueName: \"kubernetes.io/projected/0fa62dc0-5f61-4349-b7c6-9d4242503566-kube-api-access-6b7lr\") pod \"node-resolver-jzrt4\" (UID: \"0fa62dc0-5f61-4349-b7c6-9d4242503566\") " pod="openshift-dns/node-resolver-jzrt4"
Sep 16 14:35:24.371858 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.371835    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6vq\" (UniqueName: \"kubernetes.io/projected/bbc64f67-7703-4482-9ebe-fe6f420c8bc0-kube-api-access-xw6vq\") pod \"ovnkube-node-vskvj\" (UID: \"bbc64f67-7703-4482-9ebe-fe6f420c8bc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.372649 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.372533    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjm4\" (UniqueName: \"kubernetes.io/projected/96965f62-4546-4c71-89e6-d3f78095e3f9-kube-api-access-5pjm4\") pod \"iptables-alerter-wt2qd\" (UID: \"96965f62-4546-4c71-89e6-d3f78095e3f9\") " pod="openshift-network-operator/iptables-alerter-wt2qd"
Sep 16 14:35:24.372978 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.372957    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b24lj\" (UniqueName: \"kubernetes.io/projected/b80910f2-ec56-4626-bf5e-7db8a49f9765-kube-api-access-b24lj\") pod \"aws-ebs-csi-driver-node-vg67t\" (UID: \"b80910f2-ec56-4626-bf5e-7db8a49f9765\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.373362 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.373333    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gnsr\" (UniqueName: \"kubernetes.io/projected/9be55def-d862-4674-864b-04119c40ceeb-kube-api-access-5gnsr\") pod \"node-ca-vpphh\" (UID: \"9be55def-d862-4674-864b-04119c40ceeb\") " pod="openshift-image-registry/node-ca-vpphh"
Sep 16 14:35:24.374344 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.374324    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99w6l\" (UniqueName: \"kubernetes.io/projected/4e679c20-c581-4d8c-9487-e8b2ae016ec4-kube-api-access-99w6l\") pod \"multus-fzpp9\" (UID: \"4e679c20-c581-4d8c-9487-e8b2ae016ec4\") " pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.478538 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.478321    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:24.493320 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.493293    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jzrt4"
Sep 16 14:35:24.511281 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.511258    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vpphh"
Sep 16 14:35:24.524100 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.524076    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fzpp9"
Sep 16 14:35:24.530442 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.530422    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t"
Sep 16 14:35:24.559371 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.559349    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wzzx"
Sep 16 14:35:24.562187 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:24.562153    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e679c20_c581_4d8c_9487_e8b2ae016ec4.slice/crio-b1ec67f7e5d2d3f542025c77bec8372f902ea13e287219830810d0e38d5b7850 WatchSource:0}: Error finding container b1ec67f7e5d2d3f542025c77bec8372f902ea13e287219830810d0e38d5b7850: Status 404 returned error can't find the container with id b1ec67f7e5d2d3f542025c77bec8372f902ea13e287219830810d0e38d5b7850
Sep 16 14:35:24.563772 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:24.563622    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80910f2_ec56_4626_bf5e_7db8a49f9765.slice/crio-46071b6404858179ab956b5fa303b4a13fc6af9357312a28cc4e7bbfaa1b6765 WatchSource:0}: Error finding container 46071b6404858179ab956b5fa303b4a13fc6af9357312a28cc4e7bbfaa1b6765: Status 404 returned error can't find the container with id 46071b6404858179ab956b5fa303b4a13fc6af9357312a28cc4e7bbfaa1b6765
Sep 16 14:35:24.564172 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:24.564148    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99847c9ed9bfebb0b4989bf767e4cc0.slice/crio-6f8c4f9a31a0b498500048b4996b7261e3d4f97d259930177d5266ce68e5a961 WatchSource:0}: Error finding container 6f8c4f9a31a0b498500048b4996b7261e3d4f97d259930177d5266ce68e5a961: Status 404 returned error can't find the container with id 6f8c4f9a31a0b498500048b4996b7261e3d4f97d259930177d5266ce68e5a961
Sep 16 14:35:24.565804 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:24.565773    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5590f8cd50c811787920daad48d22d.slice/crio-68dd371a41430a7c4efc9168a2adb37cea2097fdd47ade677eb7e8466ee4c9e7 WatchSource:0}: Error finding container 68dd371a41430a7c4efc9168a2adb37cea2097fdd47ade677eb7e8466ee4c9e7: Status 404 returned error can't find the container with id 68dd371a41430a7c4efc9168a2adb37cea2097fdd47ade677eb7e8466ee4c9e7
Sep 16 14:35:24.568887 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.568870    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:35:24.570018 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.569996    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6pmdn"
Sep 16 14:35:24.571660 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:24.571637    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode643bf43_b88e_4f36_ae22_f0957a4864b2.slice/crio-11e503b8ae87a0190419e4a6b3f1e41f6532da95cca798a18634126ff77956d0 WatchSource:0}: Error finding container 11e503b8ae87a0190419e4a6b3f1e41f6532da95cca798a18634126ff77956d0: Status 404 returned error can't find the container with id 11e503b8ae87a0190419e4a6b3f1e41f6532da95cca798a18634126ff77956d0
Sep 16 14:35:24.579973 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:24.579949    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e4726b_a5cf_4ae6_93ff_bb145d6e1340.slice/crio-990d32f148fa84a9e4ecee85be7fde2ea712a780126e2e89165593452e3d2ea2 WatchSource:0}: Error finding container 990d32f148fa84a9e4ecee85be7fde2ea712a780126e2e89165593452e3d2ea2: Status 404 returned error can't find the container with id 990d32f148fa84a9e4ecee85be7fde2ea712a780126e2e89165593452e3d2ea2
Sep 16 14:35:24.587256 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.587218    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wt2qd"
Sep 16 14:35:24.592582 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.592565    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:24.595670 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:24.595651    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96965f62_4546_4c71_89e6_d3f78095e3f9.slice/crio-d51eba477af61593c31bbf1543d6c3d04d32449d5bc589546ec47f054d600945 WatchSource:0}: Error finding container d51eba477af61593c31bbf1543d6c3d04d32449d5bc589546ec47f054d600945: Status 404 returned error can't find the container with id d51eba477af61593c31bbf1543d6c3d04d32449d5bc589546ec47f054d600945
Sep 16 14:35:24.600932 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:24.600914    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc64f67_7703_4482_9ebe_fe6f420c8bc0.slice/crio-6780bfb9cd64af1b7c461b6e32c03c1bd8e8b7c7abb2f9dc66b2fa1277c7200c WatchSource:0}: Error finding container 6780bfb9cd64af1b7c461b6e32c03c1bd8e8b7c7abb2f9dc66b2fa1277c7200c: Status 404 returned error can't find the container with id 6780bfb9cd64af1b7c461b6e32c03c1bd8e8b7c7abb2f9dc66b2fa1277c7200c
Sep 16 14:35:24.861185 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.861089    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:24.861185 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.861140    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:24.861414 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.861252    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:24.861414 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.861269    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:24.861414 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.861303    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs podName:b52afd57-0df9-4a81-b6dc-a90212fcc659 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:25.861289769 +0000 UTC m=+3.015368035 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs") pod "network-metrics-daemon-qwq9w" (UID: "b52afd57-0df9-4a81-b6dc-a90212fcc659") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:24.861414 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.861330    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret podName:82cdd79b-96ab-48dd-a670-0d59194a7992 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:25.861309712 +0000 UTC m=+3.015387995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret") pod "global-pull-secret-syncer-r9l6v" (UID: "82cdd79b-96ab-48dd-a670-0d59194a7992") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:24.961538 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:24.961509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99ct\" (UniqueName: \"kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct\") pod \"network-check-target-n4fxc\" (UID: \"ecbbbd3b-6571-414d-94c1-130a385f4d1a\") " pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:24.961691 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.961642    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:35:24.961691 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.961656    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:35:24.961691 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.961665    2558 projected.go:196] Error preparing data for projected volume kube-api-access-t99ct for pod openshift-network-diagnostics/network-check-target-n4fxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:24.961792 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:24.961714    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct podName:ecbbbd3b-6571-414d-94c1-130a385f4d1a nodeName:}" failed. No retries permitted until 2026-09-16 14:35:25.961699354 +0000 UTC m=+3.115777623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t99ct" (UniqueName: "kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct") pod "network-check-target-n4fxc" (UID: "ecbbbd3b-6571-414d-94c1-130a385f4d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:25.072047 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:25.072005    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1641169_72fe_4cfb_b569_60d355a4f551.slice/crio-47bc3b205f9cc3532ed2c3946492504452fc55041e7f28f4a32550c0d24ac15b WatchSource:0}: Error finding container 47bc3b205f9cc3532ed2c3946492504452fc55041e7f28f4a32550c0d24ac15b: Status 404 returned error can't find the container with id 47bc3b205f9cc3532ed2c3946492504452fc55041e7f28f4a32550c0d24ac15b
Sep 16 14:35:25.106050 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:35:25.105289    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa62dc0_5f61_4349_b7c6_9d4242503566.slice/crio-3965e625c7dae96438c7db6b902c280a3fd970a63aec7ef8c6558dab6181cc8a WatchSource:0}: Error finding container 3965e625c7dae96438c7db6b902c280a3fd970a63aec7ef8c6558dab6181cc8a: Status 404 returned error can't find the container with id 3965e625c7dae96438c7db6b902c280a3fd970a63aec7ef8c6558dab6181cc8a
Sep 16 14:35:25.200383 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.200283    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 14:30:24 +0000 UTC" deadline="2028-04-20 09:06:30.752747273 +0000 UTC"
Sep 16 14:35:25.200383 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.200339    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13962h31m5.552413473s"
Sep 16 14:35:25.290477 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.290437    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:25.290665 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:25.290574    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:25.290993 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.290962    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:25.291098 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.290990    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:25.291154 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:25.291097    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:25.291232 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:25.291208    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:25.297284 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.297197    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vpphh" event={"ID":"9be55def-d862-4674-864b-04119c40ceeb","Type":"ContainerStarted","Data":"de8d5ac665d094ee4e5e3b2a35b1fc70504bd5b17639c5c5a7eea351de5b445b"}
Sep 16 14:35:25.297788 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.297639    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:35:25.299520 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.299451    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j8tbw" event={"ID":"e1641169-72fe-4cfb-b569-60d355a4f551","Type":"ContainerStarted","Data":"47bc3b205f9cc3532ed2c3946492504452fc55041e7f28f4a32550c0d24ac15b"}
Sep 16 14:35:25.302634 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.302612    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wt2qd" event={"ID":"96965f62-4546-4c71-89e6-d3f78095e3f9","Type":"ContainerStarted","Data":"d51eba477af61593c31bbf1543d6c3d04d32449d5bc589546ec47f054d600945"}
Sep 16 14:35:25.310327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.310303    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pmdn" event={"ID":"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340","Type":"ContainerStarted","Data":"990d32f148fa84a9e4ecee85be7fde2ea712a780126e2e89165593452e3d2ea2"}
Sep 16 14:35:25.317309 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.317282    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wzzx" event={"ID":"e643bf43-b88e-4f36-ae22-f0957a4864b2","Type":"ContainerStarted","Data":"11e503b8ae87a0190419e4a6b3f1e41f6532da95cca798a18634126ff77956d0"}
Sep 16 14:35:25.322579 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.322554    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-123.ec2.internal" event={"ID":"e99847c9ed9bfebb0b4989bf767e4cc0","Type":"ContainerStarted","Data":"6f8c4f9a31a0b498500048b4996b7261e3d4f97d259930177d5266ce68e5a961"}
Sep 16 14:35:25.330027 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.329998    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jzrt4" event={"ID":"0fa62dc0-5f61-4349-b7c6-9d4242503566","Type":"ContainerStarted","Data":"3965e625c7dae96438c7db6b902c280a3fd970a63aec7ef8c6558dab6181cc8a"}
Sep 16 14:35:25.333995 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.333853    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj" event={"ID":"bbc64f67-7703-4482-9ebe-fe6f420c8bc0","Type":"ContainerStarted","Data":"6780bfb9cd64af1b7c461b6e32c03c1bd8e8b7c7abb2f9dc66b2fa1277c7200c"}
Sep 16 14:35:25.338108 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.338026    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal" event={"ID":"cd5590f8cd50c811787920daad48d22d","Type":"ContainerStarted","Data":"68dd371a41430a7c4efc9168a2adb37cea2097fdd47ade677eb7e8466ee4c9e7"}
Sep 16 14:35:25.348764 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.348731    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t" event={"ID":"b80910f2-ec56-4626-bf5e-7db8a49f9765","Type":"ContainerStarted","Data":"46071b6404858179ab956b5fa303b4a13fc6af9357312a28cc4e7bbfaa1b6765"}
Sep 16 14:35:25.359748 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.359702    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fzpp9" event={"ID":"4e679c20-c581-4d8c-9487-e8b2ae016ec4","Type":"ContainerStarted","Data":"b1ec67f7e5d2d3f542025c77bec8372f902ea13e287219830810d0e38d5b7850"}
Sep 16 14:35:25.870170 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.870125    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:25.870386 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.870222    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:25.870386 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:25.870359    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:25.870517 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:25.870426    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret podName:82cdd79b-96ab-48dd-a670-0d59194a7992 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:27.870407143 +0000 UTC m=+5.024485432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret") pod "global-pull-secret-syncer-r9l6v" (UID: "82cdd79b-96ab-48dd-a670-0d59194a7992") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:25.870826 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:25.870805    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:25.870926 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:25.870862    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs podName:b52afd57-0df9-4a81-b6dc-a90212fcc659 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:27.87084615 +0000 UTC m=+5.024924432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs") pod "network-metrics-daemon-qwq9w" (UID: "b52afd57-0df9-4a81-b6dc-a90212fcc659") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:25.971096 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:25.971056    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99ct\" (UniqueName: \"kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct\") pod \"network-check-target-n4fxc\" (UID: \"ecbbbd3b-6571-414d-94c1-130a385f4d1a\") " pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:25.971301 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:25.971284    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:35:25.971387 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:25.971311    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:35:25.971387 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:25.971324    2558 projected.go:196] Error preparing data for projected volume kube-api-access-t99ct for pod openshift-network-diagnostics/network-check-target-n4fxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:25.971500 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:25.971387    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct podName:ecbbbd3b-6571-414d-94c1-130a385f4d1a nodeName:}" failed. No retries permitted until 2026-09-16 14:35:27.971368051 +0000 UTC m=+5.125446338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t99ct" (UniqueName: "kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct") pod "network-check-target-n4fxc" (UID: "ecbbbd3b-6571-414d-94c1-130a385f4d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:26.201144 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:26.201026    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 14:30:24 +0000 UTC" deadline="2028-04-03 00:27:36.298706653 +0000 UTC"
Sep 16 14:35:26.201144 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:26.201093    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13545h52m10.097618716s"
Sep 16 14:35:26.464048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:26.463753    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:35:26.885147 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:26.885067    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 14:35:27.291550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:27.291474    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:27.292091 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:27.291605    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:27.292091 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:27.292037    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:27.292218 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:27.292182    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:27.292284 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:27.292273    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:27.292388 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:27.292364    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:27.891173 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:27.890310    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:27.891173 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:27.890383    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:27.891173 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:27.890526    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:27.891173 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:27.890592    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret podName:82cdd79b-96ab-48dd-a670-0d59194a7992 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:31.890571488 +0000 UTC m=+9.044649774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret") pod "global-pull-secret-syncer-r9l6v" (UID: "82cdd79b-96ab-48dd-a670-0d59194a7992") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:27.891173 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:27.890986    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:27.891173 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:27.891035    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs podName:b52afd57-0df9-4a81-b6dc-a90212fcc659 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:31.891019778 +0000 UTC m=+9.045098060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs") pod "network-metrics-daemon-qwq9w" (UID: "b52afd57-0df9-4a81-b6dc-a90212fcc659") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:27.992976 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:27.992938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99ct\" (UniqueName: \"kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct\") pod \"network-check-target-n4fxc\" (UID: \"ecbbbd3b-6571-414d-94c1-130a385f4d1a\") " pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:27.993198 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:27.993145    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:35:27.993198 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:27.993194    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:35:27.993326 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:27.993210    2558 projected.go:196] Error preparing data for projected volume kube-api-access-t99ct for pod openshift-network-diagnostics/network-check-target-n4fxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:27.993326 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:27.993283    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct podName:ecbbbd3b-6571-414d-94c1-130a385f4d1a nodeName:}" failed. No retries permitted until 2026-09-16 14:35:31.993264402 +0000 UTC m=+9.147342683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t99ct" (UniqueName: "kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct") pod "network-check-target-n4fxc" (UID: "ecbbbd3b-6571-414d-94c1-130a385f4d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:29.289580 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:29.289540    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:29.290239 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:29.289684    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:29.290439 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:29.290268    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:29.290439 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:29.290378    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:29.290562 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:29.290516    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:29.290638 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:29.290615    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:31.291271 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:31.291243    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:31.291708 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:31.291364    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:31.291772 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:31.291749    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:31.292051 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:31.291832    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:31.292051 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:31.291892    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:31.292051 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:31.291969    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:31.925289 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:31.925249    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:31.925493 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:31.925326    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:31.925493 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:31.925447    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:31.925598 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:31.925541    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs podName:b52afd57-0df9-4a81-b6dc-a90212fcc659 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:39.925518097 +0000 UTC m=+17.079596365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs") pod "network-metrics-daemon-qwq9w" (UID: "b52afd57-0df9-4a81-b6dc-a90212fcc659") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:31.925670 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:31.925455    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:31.925732 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:31.925723    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret podName:82cdd79b-96ab-48dd-a670-0d59194a7992 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:39.92570964 +0000 UTC m=+17.079787914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret") pod "global-pull-secret-syncer-r9l6v" (UID: "82cdd79b-96ab-48dd-a670-0d59194a7992") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:32.026945 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:32.026363    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99ct\" (UniqueName: \"kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct\") pod \"network-check-target-n4fxc\" (UID: \"ecbbbd3b-6571-414d-94c1-130a385f4d1a\") " pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:32.026945 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:32.026514    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:35:32.026945 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:32.026535    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:35:32.026945 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:32.026547    2558 projected.go:196] Error preparing data for projected volume kube-api-access-t99ct for pod openshift-network-diagnostics/network-check-target-n4fxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:32.026945 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:32.026609    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct podName:ecbbbd3b-6571-414d-94c1-130a385f4d1a nodeName:}" failed. No retries permitted until 2026-09-16 14:35:40.026591617 +0000 UTC m=+17.180669901 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t99ct" (UniqueName: "kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct") pod "network-check-target-n4fxc" (UID: "ecbbbd3b-6571-414d-94c1-130a385f4d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:33.290784 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:33.290255    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:33.290784 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:33.290358    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:33.290784 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:33.290495    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:33.290784 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:33.290587    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:33.290784 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:33.290625    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:33.290784 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:33.290692    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:35.289639 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:35.289607    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:35.290061 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:35.289641    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:35.290061 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:35.289607    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:35.290061 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:35.289731    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:35.290061 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:35.289825    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:35.290061 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:35.289894    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:37.289670 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:37.289632    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:37.290092 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:37.289634    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:37.290092 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:37.289761    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:37.290092 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:37.289775    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:37.290092 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:37.289887    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:37.290092 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:37.289980    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:39.289454 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:39.289420    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:39.289844 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:39.289463    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:39.289844 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:39.289431    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:39.289844 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:39.289547    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:39.289844 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:39.289703    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:39.289844 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:39.289788    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:39.977133 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:39.977077    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:39.977390 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:39.977238    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:39.977390 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:39.977255    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:39.977390 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:39.977338    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret podName:82cdd79b-96ab-48dd-a670-0d59194a7992 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:55.977322563 +0000 UTC m=+33.131400828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret") pod "global-pull-secret-syncer-r9l6v" (UID: "82cdd79b-96ab-48dd-a670-0d59194a7992") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:39.977390 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:39.977363    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:39.977615 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:39.977419    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs podName:b52afd57-0df9-4a81-b6dc-a90212fcc659 nodeName:}" failed. No retries permitted until 2026-09-16 14:35:55.977401591 +0000 UTC m=+33.131479859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs") pod "network-metrics-daemon-qwq9w" (UID: "b52afd57-0df9-4a81-b6dc-a90212fcc659") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:40.078034 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:40.077995    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99ct\" (UniqueName: \"kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct\") pod \"network-check-target-n4fxc\" (UID: \"ecbbbd3b-6571-414d-94c1-130a385f4d1a\") " pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:40.078243 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:40.078156    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:35:40.078243 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:40.078196    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:35:40.078243 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:40.078207    2558 projected.go:196] Error preparing data for projected volume kube-api-access-t99ct for pod openshift-network-diagnostics/network-check-target-n4fxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:40.078408 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:40.078274    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct podName:ecbbbd3b-6571-414d-94c1-130a385f4d1a nodeName:}" failed. No retries permitted until 2026-09-16 14:35:56.078253441 +0000 UTC m=+33.232331706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t99ct" (UniqueName: "kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct") pod "network-check-target-n4fxc" (UID: "ecbbbd3b-6571-414d-94c1-130a385f4d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:41.292430 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:41.292400    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:41.293053 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:41.292508    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:41.293053 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:41.292515    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:41.293053 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:41.292600    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:41.293053 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:41.292649    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:41.293053 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:41.292726    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:43.290157 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:43.290092    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:43.290575 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:43.290219    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:43.290575 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:43.290226    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:43.290575 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:43.290333    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:43.290575 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:43.290400    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:43.290575 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:43.290491    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:45.289697 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.289662    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:45.289697 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.289677    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:45.289697 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.289678    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:45.290526 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:45.289762    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:45.290526 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:45.289866    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:45.290526 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:45.289970    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:45.416491 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.416456    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jzrt4" event={"ID":"0fa62dc0-5f61-4349-b7c6-9d4242503566","Type":"ContainerStarted","Data":"1ab7ac0c5889a21a6f1cea421d966a76134cca20fe9d76f6af0f7495f0485205"}
Sep 16 14:35:45.419023 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.418972    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:35:45.419282 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.419264    2558 generic.go:356] "Generic (PLEG): container finished" podID="bbc64f67-7703-4482-9ebe-fe6f420c8bc0" containerID="251ecc50b619975568814fae532cc7d23050ad99d30e470931d9f3024086d49c" exitCode=1
Sep 16 14:35:45.419340 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.419324    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj" event={"ID":"bbc64f67-7703-4482-9ebe-fe6f420c8bc0","Type":"ContainerStarted","Data":"e4a56cc864ff74a06c2663a44e6a3efcfa46251b56bced825eacdbeef8ef635f"}
Sep 16 14:35:45.419389 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.419380    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj" event={"ID":"bbc64f67-7703-4482-9ebe-fe6f420c8bc0","Type":"ContainerStarted","Data":"42889effc9168963888c4fdcacc23bb7d6d25b57f8fdff4a2a33a75307657ffd"}
Sep 16 14:35:45.419434 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.419416    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj" event={"ID":"bbc64f67-7703-4482-9ebe-fe6f420c8bc0","Type":"ContainerStarted","Data":"f9bfde05f957cc78a1f5220e253af3c4f8efeeeac88c3c90c1d5ee31fcc4f670"}
Sep 16 14:35:45.419482 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.419449    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj" event={"ID":"bbc64f67-7703-4482-9ebe-fe6f420c8bc0","Type":"ContainerStarted","Data":"36915bae9e93d11c612b96a53f3abbf0a7dd9ddf09f967c575034c9d4af052bf"}
Sep 16 14:35:45.419528 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.419488    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj" event={"ID":"bbc64f67-7703-4482-9ebe-fe6f420c8bc0","Type":"ContainerDied","Data":"251ecc50b619975568814fae532cc7d23050ad99d30e470931d9f3024086d49c"}
Sep 16 14:35:45.419573 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.419529    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj" event={"ID":"bbc64f67-7703-4482-9ebe-fe6f420c8bc0","Type":"ContainerStarted","Data":"b32ffd2d1bd50d339cc7db9307ccf6a121998eb803df401e67320a8a08f9a09f"}
Sep 16 14:35:45.420635 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.420615    2558 generic.go:356] "Generic (PLEG): container finished" podID="cd5590f8cd50c811787920daad48d22d" containerID="f7cf42ac0b32cbc0f1427e5c364de73ca25e47842f96e7dc16d0a24a90514a3c" exitCode=0
Sep 16 14:35:45.420722 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.420669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal" event={"ID":"cd5590f8cd50c811787920daad48d22d","Type":"ContainerDied","Data":"f7cf42ac0b32cbc0f1427e5c364de73ca25e47842f96e7dc16d0a24a90514a3c"}
Sep 16 14:35:45.421885 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.421864    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t" event={"ID":"b80910f2-ec56-4626-bf5e-7db8a49f9765","Type":"ContainerStarted","Data":"fea6f6e877344c9e2f440ad9c5cb0cd3c4a1616f5cf0d16bfba0fe9c699a3a46"}
Sep 16 14:35:45.423259 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.423241    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fzpp9" event={"ID":"4e679c20-c581-4d8c-9487-e8b2ae016ec4","Type":"ContainerStarted","Data":"0c84cac88ed3660dc05466d4ec1bfacd57c77c91c54231456a1c3c5780d8072b"}
Sep 16 14:35:45.424686 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.424667    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vpphh" event={"ID":"9be55def-d862-4674-864b-04119c40ceeb","Type":"ContainerStarted","Data":"b2df1ea8a8fbd77137e5fac16fefd5877004994b44ccbb969b76a32c305b8526"}
Sep 16 14:35:45.425922 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.425903    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j8tbw" event={"ID":"e1641169-72fe-4cfb-b569-60d355a4f551","Type":"ContainerStarted","Data":"4f24739164ff534657dc8c2a576852ed93407a66338c44c94747f8f9982f17d6"}
Sep 16 14:35:45.427216 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.427195    2558 generic.go:356] "Generic (PLEG): container finished" podID="b9e4726b-a5cf-4ae6-93ff-bb145d6e1340" containerID="cf479590faf9e60f6c3a378638b620bd409452714a6780f6ab0aff4f23398770" exitCode=0
Sep 16 14:35:45.427281 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.427258    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pmdn" event={"ID":"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340","Type":"ContainerDied","Data":"cf479590faf9e60f6c3a378638b620bd409452714a6780f6ab0aff4f23398770"}
Sep 16 14:35:45.428801 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.428781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wzzx" event={"ID":"e643bf43-b88e-4f36-ae22-f0957a4864b2","Type":"ContainerStarted","Data":"b513d10b25de44db15ede8c05500640c49cf042382c5711d0d923ade85700e03"}
Sep 16 14:35:45.430099 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.430075    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-123.ec2.internal" event={"ID":"e99847c9ed9bfebb0b4989bf767e4cc0","Type":"ContainerStarted","Data":"ea2a1086b2a4bc029304bc573dd161758c422869ea15cd561cb377b209ad99c1"}
Sep 16 14:35:45.489474 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.489429    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jzrt4" podStartSLOduration=3.2380582430000002 podStartE2EDuration="22.48940982s" totalImagesPullingTime="19.251351577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:45.457699454 +0000 UTC m=+22.611777742" watchObservedRunningTime="2026-09-16 14:35:45.48940982 +0000 UTC m=+22.643488104"
Sep 16 14:35:45.530882 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.530824    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-123.ec2.internal" podStartSLOduration=21.530809646 podStartE2EDuration="21.530809646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:45.489773997 +0000 UTC m=+22.643852282" watchObservedRunningTime="2026-09-16 14:35:45.530809646 +0000 UTC m=+22.684887932"
Sep 16 14:35:45.558303 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.558256    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j8tbw" podStartSLOduration=7.192366701 podStartE2EDuration="22.558240306s" totalImagesPullingTime="15.365873605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:45.531110017 +0000 UTC m=+22.685188306" watchObservedRunningTime="2026-09-16 14:35:45.558240306 +0000 UTC m=+22.712318592"
Sep 16 14:35:45.558442 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.558387    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vpphh" podStartSLOduration=3.379545064 podStartE2EDuration="22.558380741s" totalImagesPullingTime="19.178835677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:45.558187539 +0000 UTC m=+22.712265827" watchObservedRunningTime="2026-09-16 14:35:45.558380741 +0000 UTC m=+22.712459074"
Sep 16 14:35:45.642948 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.642874    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fzpp9" podStartSLOduration=2.701047396 podStartE2EDuration="22.642820408s" totalImagesPullingTime="19.941773012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:45.591409934 +0000 UTC m=+22.745488233" watchObservedRunningTime="2026-09-16 14:35:45.642820408 +0000 UTC m=+22.796898697"
Sep 16 14:35:45.643292 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:45.643257    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4wzzx" podStartSLOduration=2.837648032 podStartE2EDuration="22.643243563s" totalImagesPullingTime="19.805595531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:45.643230589 +0000 UTC m=+22.797308878" watchObservedRunningTime="2026-09-16 14:35:45.643243563 +0000 UTC m=+22.797321850"
Sep 16 14:35:46.278333 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:46.278307    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 14:35:46.434386 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:46.434297    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wt2qd" event={"ID":"96965f62-4546-4c71-89e6-d3f78095e3f9","Type":"ContainerStarted","Data":"47633d582cbd8b09dface0584b6edd319e2cee09f8ea586c82782c6bca0d7d90"}
Sep 16 14:35:46.436562 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:46.436526    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t" event={"ID":"b80910f2-ec56-4626-bf5e-7db8a49f9765","Type":"ContainerStarted","Data":"3e3c911468b2ffa942c04b92e899993be4d330cddcf84692cf804574cb3daf04"}
Sep 16 14:35:46.496607 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:46.496555    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wt2qd" podStartSLOduration=3.735603866 podStartE2EDuration="23.49653268s" totalImagesPullingTime="19.760928814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:46.460806391 +0000 UTC m=+23.614884679" watchObservedRunningTime="2026-09-16 14:35:46.49653268 +0000 UTC m=+23.650610969"
Sep 16 14:35:46.893372 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:46.893336    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:46.893912 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:46.893890    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:47.220036 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.219921    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T14:35:46.278326044Z","UUID":"be68772d-1ab3-410e-85d2-e2349aaee3f9","Handler":null,"Name":"","Endpoint":""}
Sep 16 14:35:47.221520 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.221501    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 14:35:47.221640 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.221531    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 14:35:47.289584 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.289553    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:47.289764 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.289679    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:47.289764 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:47.289693    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:47.289882 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:47.289808    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:47.289882 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.289860    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:47.289984 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:47.289926    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:47.441151 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.441122    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal" event={"ID":"cd5590f8cd50c811787920daad48d22d","Type":"ContainerStarted","Data":"6d8f728b4a496bcbc38f669c4a54f9685ef0aeffebc592467663b785041937bc"}
Sep 16 14:35:47.444862 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.444843    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:35:47.445245 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.445216    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj" event={"ID":"bbc64f67-7703-4482-9ebe-fe6f420c8bc0","Type":"ContainerStarted","Data":"3970f2846d32e03ef9d96cdcf1420080ecb921e21541cda34b319150b5fc7d07"}
Sep 16 14:35:47.445659 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.445633    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:47.446160 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.446139    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j8tbw"
Sep 16 14:35:47.476123 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:47.476077    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-123.ec2.internal" podStartSLOduration=23.476057733 podStartE2EDuration="23.476057733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:47.474581942 +0000 UTC m=+24.628660229" watchObservedRunningTime="2026-09-16 14:35:47.476057733 +0000 UTC m=+24.630136023"
Sep 16 14:35:48.449910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:48.449876    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t" event={"ID":"b80910f2-ec56-4626-bf5e-7db8a49f9765","Type":"ContainerStarted","Data":"d9e6b8a6cf2489db9f21241b946fdaca286510e17aae6eae0a64c2d8c4b3de88"}
Sep 16 14:35:48.478392 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:48.478347    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vg67t" podStartSLOduration=3.32386285 podStartE2EDuration="25.478330802s" totalImagesPullingTime="22.154467952s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:48.47700406 +0000 UTC m=+25.631082344" watchObservedRunningTime="2026-09-16 14:35:48.478330802 +0000 UTC m=+25.632409093"
Sep 16 14:35:49.289211 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:49.289176    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:49.289384 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:49.289161    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:49.289384 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:49.289304    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:49.289384 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:49.289161    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:49.289526 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:49.289392    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:49.289526 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:49.289437    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:50.458576 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:50.458306    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:35:50.459527 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:50.458679    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj" event={"ID":"bbc64f67-7703-4482-9ebe-fe6f420c8bc0","Type":"ContainerStarted","Data":"baad6a62df19eeaf57719b92c580036c36185bd269cee68bce96314b86ae4534"}
Sep 16 14:35:50.459527 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:50.458951    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:50.459527 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:50.459030    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:50.459527 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:50.459191    2558 scope.go:117] "RemoveContainer" containerID="251ecc50b619975568814fae532cc7d23050ad99d30e470931d9f3024086d49c"
Sep 16 14:35:50.476906 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:50.476757    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:51.289436 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:51.289404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:51.289634 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:51.289404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:51.289634 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:51.289500    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:51.289634 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:51.289414    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:51.289798 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:51.289570    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:51.289798 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:51.289668    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:51.462749 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:51.462715    2558 generic.go:356] "Generic (PLEG): container finished" podID="b9e4726b-a5cf-4ae6-93ff-bb145d6e1340" containerID="7b504d4aa5e4963e935d83a3d953e926adca76b681a65bec2d439c35fbf90a33" exitCode=0
Sep 16 14:35:51.463190 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:51.462794    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pmdn" event={"ID":"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340","Type":"ContainerDied","Data":"7b504d4aa5e4963e935d83a3d953e926adca76b681a65bec2d439c35fbf90a33"}
Sep 16 14:35:51.466659 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:51.466643    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:35:51.467023 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:51.467004    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj" event={"ID":"bbc64f67-7703-4482-9ebe-fe6f420c8bc0","Type":"ContainerStarted","Data":"cd6faaa959cd6b23ce5586260d0671f5b90ed03bc2ed6ba43e6d3abad2bef6ab"}
Sep 16 14:35:51.467226 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:51.467206    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:51.481970 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:51.481950    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:35:51.527621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:51.527573    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj" podStartSLOduration=8.718233078 podStartE2EDuration="28.527558569s" totalImagesPullingTime="19.809325491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:35:51.527248357 +0000 UTC m=+28.681326643" watchObservedRunningTime="2026-09-16 14:35:51.527558569 +0000 UTC m=+28.681636855"
Sep 16 14:35:51.923148 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:51.923120    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jzrt4_0fa62dc0-5f61-4349-b7c6-9d4242503566/dns-node-resolver/0.log"
Sep 16 14:35:52.428008 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:52.427979    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qwq9w"]
Sep 16 14:35:52.428202 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:52.428117    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:52.428254 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:52.428219    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:52.436771 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:52.436742    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n4fxc"]
Sep 16 14:35:52.436895 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:52.436846    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:52.436965 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:52.436945    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:52.437375 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:52.437346    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r9l6v"]
Sep 16 14:35:52.437491 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:52.437454    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:52.437571 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:52.437550    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:53.093072 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:53.092919    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vpphh_9be55def-d862-4674-864b-04119c40ceeb/node-ca/0.log"
Sep 16 14:35:53.473304 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:53.473272    2558 generic.go:356] "Generic (PLEG): container finished" podID="b9e4726b-a5cf-4ae6-93ff-bb145d6e1340" containerID="8facc97f2fcf19de49b401fb6786dad0418096780158e76eb5c6d9b5a752677c" exitCode=0
Sep 16 14:35:53.473464 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:53.473354    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pmdn" event={"ID":"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340","Type":"ContainerDied","Data":"8facc97f2fcf19de49b401fb6786dad0418096780158e76eb5c6d9b5a752677c"}
Sep 16 14:35:54.288970 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:54.288941    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:54.289365 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:54.289082    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:54.289365 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:54.289095    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:54.289365 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:54.289119    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:54.289365 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:54.289266    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:54.289365 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:54.289333    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:55.480067 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:55.480038    2558 generic.go:356] "Generic (PLEG): container finished" podID="b9e4726b-a5cf-4ae6-93ff-bb145d6e1340" containerID="e25616400fb42123347018ccca3906ceb66ecc6ccd720d96c21bf2292969384e" exitCode=0
Sep 16 14:35:55.480428 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:55.480076    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pmdn" event={"ID":"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340","Type":"ContainerDied","Data":"e25616400fb42123347018ccca3906ceb66ecc6ccd720d96c21bf2292969384e"}
Sep 16 14:35:55.992181 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:55.992139    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:55.992346 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:55.992192    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:55.992346 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:55.992277    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:55.992346 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:55.992280    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:55.992346 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:55.992324    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret podName:82cdd79b-96ab-48dd-a670-0d59194a7992 nodeName:}" failed. No retries permitted until 2026-09-16 14:36:27.992310936 +0000 UTC m=+65.146389201 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret") pod "global-pull-secret-syncer-r9l6v" (UID: "82cdd79b-96ab-48dd-a670-0d59194a7992") : object "kube-system"/"original-pull-secret" not registered
Sep 16 14:35:55.992346 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:55.992336    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs podName:b52afd57-0df9-4a81-b6dc-a90212fcc659 nodeName:}" failed. No retries permitted until 2026-09-16 14:36:27.992330676 +0000 UTC m=+65.146408942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs") pod "network-metrics-daemon-qwq9w" (UID: "b52afd57-0df9-4a81-b6dc-a90212fcc659") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 14:35:56.092632 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:56.092605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99ct\" (UniqueName: \"kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct\") pod \"network-check-target-n4fxc\" (UID: \"ecbbbd3b-6571-414d-94c1-130a385f4d1a\") " pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:56.092786 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:56.092749    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 14:35:56.092786 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:56.092765    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 14:35:56.092786 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:56.092774    2558 projected.go:196] Error preparing data for projected volume kube-api-access-t99ct for pod openshift-network-diagnostics/network-check-target-n4fxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:56.092941 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:56.092826    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct podName:ecbbbd3b-6571-414d-94c1-130a385f4d1a nodeName:}" failed. No retries permitted until 2026-09-16 14:36:28.092809826 +0000 UTC m=+65.246888108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t99ct" (UniqueName: "kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct") pod "network-check-target-n4fxc" (UID: "ecbbbd3b-6571-414d-94c1-130a385f4d1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 14:35:56.289646 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:56.289563    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:56.289646 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:56.289603    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:56.289819 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:56.289671    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:56.289819 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:56.289686    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:56.289819 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:56.289751    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:56.289819 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:56.289803    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:35:58.289362 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:58.289329    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:35:58.289802 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:58.289329    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:35:58.289802 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:58.289428    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:35:58.289802 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:35:58.289329    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:35:58.289802 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:58.289545    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:35:58.289802 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:35:58.289667    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:36:00.289822 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:00.289786    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:00.289822 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:00.289806    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:00.289822 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:00.289811    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:00.290323 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:00.289946    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:36:00.290323 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:00.290042    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:36:00.290323 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:00.290123    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:36:01.496457 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:01.496257    2558 generic.go:356] "Generic (PLEG): container finished" podID="b9e4726b-a5cf-4ae6-93ff-bb145d6e1340" containerID="dff4fef96a770b9dc202f4dd23d644dc0ae97b8f215eb9bcf821a1489a16d646" exitCode=0
Sep 16 14:36:01.496457 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:01.496329    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pmdn" event={"ID":"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340","Type":"ContainerDied","Data":"dff4fef96a770b9dc202f4dd23d644dc0ae97b8f215eb9bcf821a1489a16d646"}
Sep 16 14:36:02.289240 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:02.289157    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:02.289382 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:02.289245    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:02.289382 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:02.289258    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:36:02.289382 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:02.289157    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:02.289382 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:02.289328    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:36:02.289521 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:02.289425    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:36:02.502037 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:02.502006    2558 generic.go:356] "Generic (PLEG): container finished" podID="b9e4726b-a5cf-4ae6-93ff-bb145d6e1340" containerID="2703fc0bb94104ad902db76005b716cda4fcddfec8de8fca0c2cadb13a964b48" exitCode=0
Sep 16 14:36:02.502413 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:02.502065    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pmdn" event={"ID":"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340","Type":"ContainerDied","Data":"2703fc0bb94104ad902db76005b716cda4fcddfec8de8fca0c2cadb13a964b48"}
Sep 16 14:36:03.507419 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:03.507383    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pmdn" event={"ID":"b9e4726b-a5cf-4ae6-93ff-bb145d6e1340","Type":"ContainerStarted","Data":"780ce61f3ad6a332d165a7defd742f240f5ae260ef04bcb20490f4c2ffd619c1"}
Sep 16 14:36:03.568466 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:03.568418    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6pmdn" podStartSLOduration=8.0281289 podStartE2EDuration="40.568404616s" totalImagesPullingTime="32.540275716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:03.568206272 +0000 UTC m=+40.722284562" watchObservedRunningTime="2026-09-16 14:36:03.568404616 +0000 UTC m=+40.722482934"
Sep 16 14:36:04.289720 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:04.289689    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:04.289720 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:04.289713    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:04.289917 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:04.289787    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:04.289917 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:04.289808    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:36:04.289917 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:04.289867    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:36:04.290024 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:04.289928    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:36:06.289631 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:06.289601    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:06.289993 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:06.289699    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:06.289993 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:06.289709    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:36:06.289993 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:06.289725    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:06.289993 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:06.289796    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:36:06.289993 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:06.289852    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:36:08.289221 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:08.289181    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:08.289221 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:08.289210    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:08.289696 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:08.289196    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:08.289696 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:08.289301    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:36:08.289696 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:08.289376    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:36:08.289696 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:08.289425    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:36:10.289400 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:10.289362    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:10.289865 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:10.289362    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:10.289865 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:10.289494    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:36:10.289865 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:10.289362    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:10.289865 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:10.289533    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:36:10.289865 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:10.289603    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:36:12.289406 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:12.289372    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:12.289406 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:12.289397    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:12.289819 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:12.289469    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:36:12.289819 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:12.289373    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:12.289819 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:12.289556    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:36:12.289819 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:12.289624    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:36:14.289042 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:14.289008    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:14.289042 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:14.289032    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:14.289496 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:14.289010    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:14.289496 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:14.289097    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n4fxc" podUID="ecbbbd3b-6571-414d-94c1-130a385f4d1a"
Sep 16 14:36:14.289496 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:14.289232    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qwq9w" podUID="b52afd57-0df9-4a81-b6dc-a90212fcc659"
Sep 16 14:36:14.289496 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:14.289300    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r9l6v" podUID="82cdd79b-96ab-48dd-a670-0d59194a7992"
Sep 16 14:36:16.141584 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.141559    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-123.ec2.internal" event="NodeReady"
Sep 16 14:36:16.141950 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.141669    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 14:36:16.233068 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.233002    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zl54r"]
Sep 16 14:36:16.236409 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.236394    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zl54r"
Sep 16 14:36:16.240091 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.240068    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-69hhl"]
Sep 16 14:36:16.240765 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.240750    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 14:36:16.241012 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.240999    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8jq45\""
Sep 16 14:36:16.241476 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.241463    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 14:36:16.243471 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.243457    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 14:36:16.243980 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.243962    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.248520 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.248501    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fz4cv\""
Sep 16 14:36:16.248611 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.248552    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 14:36:16.248611 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.248559    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 14:36:16.248918 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.248803    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 14:36:16.249302 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.249287    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 14:36:16.254856 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.254833    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zl54r"]
Sep 16 14:36:16.264467 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.264442    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-69hhl"]
Sep 16 14:36:16.274743 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.274725    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wqn5h"]
Sep 16 14:36:16.278529 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.278515    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.283500 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.283485    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fbx5d\""
Sep 16 14:36:16.284197 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.284180    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 14:36:16.284285 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.284200    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 14:36:16.288874 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.288860    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:16.289026 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.289012    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:16.289121 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.289108    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:16.292552 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.292236    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 14:36:16.292552 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.292489    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 14:36:16.292552 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.292532    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sr586\""
Sep 16 14:36:16.292741 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.292535    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 14:36:16.293334 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.293128    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 14:36:16.293522 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.293507    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bf8l5\""
Sep 16 14:36:16.300469 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.300448    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wqn5h"]
Sep 16 14:36:16.332532 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.332513    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f36b8b3-e873-4e80-8755-76a5fcab21c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.332632 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.332544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f36b8b3-e873-4e80-8755-76a5fcab21c3-crio-socket\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.332632 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.332569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xx2k\" (UniqueName: \"kubernetes.io/projected/dadc893b-cc06-4432-ad74-8a14046e81fd-kube-api-access-4xx2k\") pod \"ingress-canary-zl54r\" (UID: \"dadc893b-cc06-4432-ad74-8a14046e81fd\") " pod="openshift-ingress-canary/ingress-canary-zl54r"
Sep 16 14:36:16.332632 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.332587    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rpkb\" (UniqueName: \"kubernetes.io/projected/3f36b8b3-e873-4e80-8755-76a5fcab21c3-kube-api-access-7rpkb\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.332749 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.332648    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dadc893b-cc06-4432-ad74-8a14046e81fd-cert\") pod \"ingress-canary-zl54r\" (UID: \"dadc893b-cc06-4432-ad74-8a14046e81fd\") " pod="openshift-ingress-canary/ingress-canary-zl54r"
Sep 16 14:36:16.332749 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.332673    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f36b8b3-e873-4e80-8755-76a5fcab21c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.332749 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.332699    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f36b8b3-e873-4e80-8755-76a5fcab21c3-data-volume\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.433932 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.433904    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f36b8b3-e873-4e80-8755-76a5fcab21c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.433932 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.433940    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f36b8b3-e873-4e80-8755-76a5fcab21c3-data-volume\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.434154 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f36b8b3-e873-4e80-8755-76a5fcab21c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.434154 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434125    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-metrics-tls\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.434154 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434148    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-config-volume\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.434339 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434195    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f36b8b3-e873-4e80-8755-76a5fcab21c3-crio-socket\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.434339 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xx2k\" (UniqueName: \"kubernetes.io/projected/dadc893b-cc06-4432-ad74-8a14046e81fd-kube-api-access-4xx2k\") pod \"ingress-canary-zl54r\" (UID: \"dadc893b-cc06-4432-ad74-8a14046e81fd\") " pod="openshift-ingress-canary/ingress-canary-zl54r"
Sep 16 14:36:16.434339 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434248    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdpb\" (UniqueName: \"kubernetes.io/projected/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-kube-api-access-khdpb\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.434339 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434267    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rpkb\" (UniqueName: \"kubernetes.io/projected/3f36b8b3-e873-4e80-8755-76a5fcab21c3-kube-api-access-7rpkb\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.434339 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434311    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f36b8b3-e873-4e80-8755-76a5fcab21c3-data-volume\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.434339 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434321    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-tmp-dir\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.434596 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434426    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dadc893b-cc06-4432-ad74-8a14046e81fd-cert\") pod \"ingress-canary-zl54r\" (UID: \"dadc893b-cc06-4432-ad74-8a14046e81fd\") " pod="openshift-ingress-canary/ingress-canary-zl54r"
Sep 16 14:36:16.434596 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434427    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f36b8b3-e873-4e80-8755-76a5fcab21c3-crio-socket\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.434596 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.434494    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f36b8b3-e873-4e80-8755-76a5fcab21c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.437849 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.437827    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f36b8b3-e873-4e80-8755-76a5fcab21c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.437959 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.437942    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dadc893b-cc06-4432-ad74-8a14046e81fd-cert\") pod \"ingress-canary-zl54r\" (UID: \"dadc893b-cc06-4432-ad74-8a14046e81fd\") " pod="openshift-ingress-canary/ingress-canary-zl54r"
Sep 16 14:36:16.452745 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.452717    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xx2k\" (UniqueName: \"kubernetes.io/projected/dadc893b-cc06-4432-ad74-8a14046e81fd-kube-api-access-4xx2k\") pod \"ingress-canary-zl54r\" (UID: \"dadc893b-cc06-4432-ad74-8a14046e81fd\") " pod="openshift-ingress-canary/ingress-canary-zl54r"
Sep 16 14:36:16.454649 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.454630    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rpkb\" (UniqueName: \"kubernetes.io/projected/3f36b8b3-e873-4e80-8755-76a5fcab21c3-kube-api-access-7rpkb\") pod \"insights-runtime-extractor-69hhl\" (UID: \"3f36b8b3-e873-4e80-8755-76a5fcab21c3\") " pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.535069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.534986    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-metrics-tls\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.535069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.535033    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-config-volume\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.535305 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.535079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdpb\" (UniqueName: \"kubernetes.io/projected/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-kube-api-access-khdpb\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.535305 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.535124    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-tmp-dir\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.536080 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.536059    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-tmp-dir\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.536284 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.536265    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-config-volume\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.537355 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.537337    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-metrics-tls\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.545409 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.545393    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zl54r"
Sep 16 14:36:16.546011 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.545992    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdpb\" (UniqueName: \"kubernetes.io/projected/a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985-kube-api-access-khdpb\") pod \"dns-default-wqn5h\" (UID: \"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985\") " pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.551707 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.551688    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-69hhl"
Sep 16 14:36:16.587780 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.587742    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:16.886197 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.886159    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zl54r"]
Sep 16 14:36:16.891534 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:16.891511    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadc893b_cc06_4432_ad74_8a14046e81fd.slice/crio-19fb2bf81c1914f13141c533829982d86a1b302f31619f54e7b3ac6519002ef0 WatchSource:0}: Error finding container 19fb2bf81c1914f13141c533829982d86a1b302f31619f54e7b3ac6519002ef0: Status 404 returned error can't find the container with id 19fb2bf81c1914f13141c533829982d86a1b302f31619f54e7b3ac6519002ef0
Sep 16 14:36:16.893774 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.893685    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-69hhl"]
Sep 16 14:36:16.947177 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:16.947149    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wqn5h"]
Sep 16 14:36:16.958957 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:16.958933    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85d3d0e_b3c1_40cd_90c5_6cba2ebaf985.slice/crio-b81271b5e3429d6ba48f31b677d506c770a81b191570a0fee1d6b61fd804fb9e WatchSource:0}: Error finding container b81271b5e3429d6ba48f31b677d506c770a81b191570a0fee1d6b61fd804fb9e: Status 404 returned error can't find the container with id b81271b5e3429d6ba48f31b677d506c770a81b191570a0fee1d6b61fd804fb9e
Sep 16 14:36:17.542181 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:17.542133    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zl54r" event={"ID":"dadc893b-cc06-4432-ad74-8a14046e81fd","Type":"ContainerStarted","Data":"19fb2bf81c1914f13141c533829982d86a1b302f31619f54e7b3ac6519002ef0"}
Sep 16 14:36:17.543830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:17.543796    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wqn5h" event={"ID":"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985","Type":"ContainerStarted","Data":"b81271b5e3429d6ba48f31b677d506c770a81b191570a0fee1d6b61fd804fb9e"}
Sep 16 14:36:17.546091 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:17.546029    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69hhl" event={"ID":"3f36b8b3-e873-4e80-8755-76a5fcab21c3","Type":"ContainerStarted","Data":"0d10ccc68de83123b0fb853dbaef9acdf040cedcdc09f9e7b06ffaccc32f6c16"}
Sep 16 14:36:17.546091 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:17.546085    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69hhl" event={"ID":"3f36b8b3-e873-4e80-8755-76a5fcab21c3","Type":"ContainerStarted","Data":"f469a58639e0c8775ae2457f02385bfccb6842099af512c0a948c39b5782c99d"}
Sep 16 14:36:18.550548 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:18.550509    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69hhl" event={"ID":"3f36b8b3-e873-4e80-8755-76a5fcab21c3","Type":"ContainerStarted","Data":"a9e92b15c13105015f234a5fd6ff45c128443e35251edc3be799d44571bac0cd"}
Sep 16 14:36:18.636883 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:18.636846    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d"]
Sep 16 14:36:18.640028 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:18.640002    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d"
Sep 16 14:36:18.643856 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:18.643828    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 14:36:18.643987 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:18.643946    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sg5vs\""
Sep 16 14:36:18.665578 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:18.665548    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d"]
Sep 16 14:36:18.753896 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:18.753861    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/aeace53c-a99e-40b4-8de7-743f53f5e872-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hcw2d\" (UID: \"aeace53c-a99e-40b4-8de7-743f53f5e872\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d"
Sep 16 14:36:18.854740 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:18.854703    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/aeace53c-a99e-40b4-8de7-743f53f5e872-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hcw2d\" (UID: \"aeace53c-a99e-40b4-8de7-743f53f5e872\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d"
Sep 16 14:36:18.857396 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:18.857371    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/aeace53c-a99e-40b4-8de7-743f53f5e872-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-hcw2d\" (UID: \"aeace53c-a99e-40b4-8de7-743f53f5e872\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d"
Sep 16 14:36:18.951183 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:18.951119    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d"
Sep 16 14:36:19.557055 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:19.556542    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zl54r" event={"ID":"dadc893b-cc06-4432-ad74-8a14046e81fd","Type":"ContainerStarted","Data":"80b9a3080f585b8c0594868bea73fabc84d76de22649d7ceada7904e2f64d3d8"}
Sep 16 14:36:19.560275 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:19.560213    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wqn5h" event={"ID":"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985","Type":"ContainerStarted","Data":"9129b02e5591f62745265fbc053e088e0b93b38a72308b693959b0654914bdb4"}
Sep 16 14:36:19.720333 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:19.719640    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zl54r" podStartSLOduration=1.176220748 podStartE2EDuration="3.719615889s" totalImagesPullingTime="2.543395141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:19.597855031 +0000 UTC m=+56.751933320" watchObservedRunningTime="2026-09-16 14:36:19.719615889 +0000 UTC m=+56.873694176"
Sep 16 14:36:19.720333 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:19.720225    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d"]
Sep 16 14:36:19.723929 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:19.723898    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeace53c_a99e_40b4_8de7_743f53f5e872.slice/crio-0b98eb93dec3db49b7821005d5a5e05ffe010e929e600f405a632d46b6f83cf9 WatchSource:0}: Error finding container 0b98eb93dec3db49b7821005d5a5e05ffe010e929e600f405a632d46b6f83cf9: Status 404 returned error can't find the container with id 0b98eb93dec3db49b7821005d5a5e05ffe010e929e600f405a632d46b6f83cf9
Sep 16 14:36:20.565875 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:20.565838    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wqn5h" event={"ID":"a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985","Type":"ContainerStarted","Data":"8af3b682e235f9d6380a48e0b29581fbdf91cc049aa8d52a8b56aaa9c700c741"}
Sep 16 14:36:20.566315 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:20.565935    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:20.567546 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:20.567521    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d" event={"ID":"aeace53c-a99e-40b4-8de7-743f53f5e872","Type":"ContainerStarted","Data":"0b98eb93dec3db49b7821005d5a5e05ffe010e929e600f405a632d46b6f83cf9"}
Sep 16 14:36:20.569839 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:20.569805    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69hhl" event={"ID":"3f36b8b3-e873-4e80-8755-76a5fcab21c3","Type":"ContainerStarted","Data":"0d06ace9f8673cf723ac2b766e137e438c0c5bf52a724701771ee70bd5a3ac17"}
Sep 16 14:36:20.601418 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:20.601355    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wqn5h" podStartSLOduration=2.131806904 podStartE2EDuration="4.601336455s" totalImagesPullingTime="2.469529551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:20.600413183 +0000 UTC m=+57.754491471" watchObservedRunningTime="2026-09-16 14:36:20.601336455 +0000 UTC m=+57.755414740"
Sep 16 14:36:20.625487 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:20.625322    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-69hhl" podStartSLOduration=1.2997836409999999 podStartE2EDuration="4.625304429s" totalImagesPullingTime="3.325520788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:36:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:20.625108296 +0000 UTC m=+57.779186586" watchObservedRunningTime="2026-09-16 14:36:20.625304429 +0000 UTC m=+57.779382715"
Sep 16 14:36:21.574553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:21.574512    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d" event={"ID":"aeace53c-a99e-40b4-8de7-743f53f5e872","Type":"ContainerStarted","Data":"1fe97b260ec3dc6f64d1981342c3cdd57e60dab64ba891d0f839ea7bac633b4a"}
Sep 16 14:36:21.574982 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:21.574725    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d"
Sep 16 14:36:21.579434 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:21.579408    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d"
Sep 16 14:36:21.605516 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:21.605480    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-hcw2d" podStartSLOduration=2.279814209 podStartE2EDuration="3.605466239s" totalImagesPullingTime="1.32565203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:21.605440665 +0000 UTC m=+58.759518952" watchObservedRunningTime="2026-09-16 14:36:21.605466239 +0000 UTC m=+58.759544517"
Sep 16 14:36:22.679943 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.679914    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"]
Sep 16 14:36:22.710800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.710776    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"]
Sep 16 14:36:22.710947 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.710883    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:22.715525 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.715308    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 14:36:22.715525 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.715341    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 14:36:22.715525 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.715378    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 14:36:22.715525 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.715404    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 14:36:22.715525 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.715378    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 14:36:22.715525 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.715378    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fff5f\""
Sep 16 14:36:22.885024 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.884986    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d8253c9-5d09-4990-aedc-7072f4b53d1c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:22.885185 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.885034    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d8253c9-5d09-4990-aedc-7072f4b53d1c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:22.885185 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.885113    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvh9w\" (UniqueName: \"kubernetes.io/projected/1d8253c9-5d09-4990-aedc-7072f4b53d1c-kube-api-access-rvh9w\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:22.885185 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.885145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d8253c9-5d09-4990-aedc-7072f4b53d1c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:22.986185 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.986084    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d8253c9-5d09-4990-aedc-7072f4b53d1c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:22.986185 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.986139    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d8253c9-5d09-4990-aedc-7072f4b53d1c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:22.986386 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.986223    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvh9w\" (UniqueName: \"kubernetes.io/projected/1d8253c9-5d09-4990-aedc-7072f4b53d1c-kube-api-access-rvh9w\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:22.986386 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:22.986240    2558 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 16 14:36:22.986386 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.986261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d8253c9-5d09-4990-aedc-7072f4b53d1c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:22.986386 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:22.986299    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8253c9-5d09-4990-aedc-7072f4b53d1c-prometheus-operator-tls podName:1d8253c9-5d09-4990-aedc-7072f4b53d1c nodeName:}" failed. No retries permitted until 2026-09-16 14:36:23.486284004 +0000 UTC m=+60.640362269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1d8253c9-5d09-4990-aedc-7072f4b53d1c-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-fqk9w" (UID: "1d8253c9-5d09-4990-aedc-7072f4b53d1c") : secret "prometheus-operator-tls" not found
Sep 16 14:36:22.986833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.986810    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d8253c9-5d09-4990-aedc-7072f4b53d1c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:22.988559 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.988537    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d8253c9-5d09-4990-aedc-7072f4b53d1c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:22.995510 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:22.995489    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvh9w\" (UniqueName: \"kubernetes.io/projected/1d8253c9-5d09-4990-aedc-7072f4b53d1c-kube-api-access-rvh9w\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:23.488249 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:23.488219    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d8253c9-5d09-4990-aedc-7072f4b53d1c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:23.488844 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:23.488824    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vskvj"
Sep 16 14:36:23.490320 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:23.490301    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d8253c9-5d09-4990-aedc-7072f4b53d1c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-fqk9w\" (UID: \"1d8253c9-5d09-4990-aedc-7072f4b53d1c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:23.621715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:23.621687    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fff5f\""
Sep 16 14:36:23.629271 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:23.629252    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"
Sep 16 14:36:23.914823 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:23.914800    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w"]
Sep 16 14:36:23.916182 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:23.916141    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8253c9_5d09_4990_aedc_7072f4b53d1c.slice/crio-9346df20713b6bb8b8b0ab20b20181de0751cf8671059228b9b911f945fc5979 WatchSource:0}: Error finding container 9346df20713b6bb8b8b0ab20b20181de0751cf8671059228b9b911f945fc5979: Status 404 returned error can't find the container with id 9346df20713b6bb8b8b0ab20b20181de0751cf8671059228b9b911f945fc5979
Sep 16 14:36:24.042920 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.042892    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-546568fdb5-cl296"]
Sep 16 14:36:24.063048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.063022    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546568fdb5-cl296"]
Sep 16 14:36:24.063157 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.063136    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.066223 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.066204    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b2vv8\""
Sep 16 14:36:24.066329 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.066293    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 14:36:24.066395 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.066343    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 14:36:24.066448 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.066407    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 14:36:24.067341 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.067326    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 14:36:24.067432 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.067392    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 14:36:24.067486 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.067395    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 14:36:24.067663 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.067647    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 14:36:24.091560 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.091537    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-serving-cert\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.091700 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.091584    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-config\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.091700 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.091601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-oauth-serving-cert\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.091700 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.091685    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd4vf\" (UniqueName: \"kubernetes.io/projected/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-kube-api-access-rd4vf\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.091865 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.091718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-service-ca\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.091865 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.091750    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-oauth-config\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.192677 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.192613    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-service-ca\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.192677 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.192649    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-oauth-config\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.192874 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.192715    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-serving-cert\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.192874 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.192773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-config\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.192874 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.192796    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-oauth-serving-cert\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.192874 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.192826    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4vf\" (UniqueName: \"kubernetes.io/projected/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-kube-api-access-rd4vf\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.193400 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.193378    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-service-ca\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.193516 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.193459    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-config\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.193516 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.193510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-oauth-serving-cert\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.195036 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.195019    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-oauth-config\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.195154 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.195140    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-serving-cert\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.202310 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.202293    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd4vf\" (UniqueName: \"kubernetes.io/projected/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-kube-api-access-rd4vf\") pod \"console-546568fdb5-cl296\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") " pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.372025 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.371987    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:24.589626 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.589585    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w" event={"ID":"1d8253c9-5d09-4990-aedc-7072f4b53d1c","Type":"ContainerStarted","Data":"9346df20713b6bb8b8b0ab20b20181de0751cf8671059228b9b911f945fc5979"}
Sep 16 14:36:24.643764 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:24.643732    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-546568fdb5-cl296"]
Sep 16 14:36:24.647665 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:24.647630    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f5bc1e_e8b1_49d3_acf2_4a8e9dd33146.slice/crio-4413e478d188f3bdcc8785e30020d24eb246bb2a89d9bb2e117973a1f688833d WatchSource:0}: Error finding container 4413e478d188f3bdcc8785e30020d24eb246bb2a89d9bb2e117973a1f688833d: Status 404 returned error can't find the container with id 4413e478d188f3bdcc8785e30020d24eb246bb2a89d9bb2e117973a1f688833d
Sep 16 14:36:25.595185 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:25.595132    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w" event={"ID":"1d8253c9-5d09-4990-aedc-7072f4b53d1c","Type":"ContainerStarted","Data":"3b55848866a12a4e0a1adbb64873dc0cfccbd4310b426976d6d3e6a67a914f49"}
Sep 16 14:36:25.595651 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:25.595216    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w" event={"ID":"1d8253c9-5d09-4990-aedc-7072f4b53d1c","Type":"ContainerStarted","Data":"4a49812dba1495925a350d1993848edce021928497ee66d9b9faaf8917e60010"}
Sep 16 14:36:25.596559 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:25.596532    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546568fdb5-cl296" event={"ID":"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146","Type":"ContainerStarted","Data":"4413e478d188f3bdcc8785e30020d24eb246bb2a89d9bb2e117973a1f688833d"}
Sep 16 14:36:25.616556 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:25.616506    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-fqk9w" podStartSLOduration=2.373107638 podStartE2EDuration="3.616485152s" totalImagesPullingTime="1.243377514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:25.615189265 +0000 UTC m=+62.769267551" watchObservedRunningTime="2026-09-16 14:36:25.616485152 +0000 UTC m=+62.770563444"
Sep 16 14:36:27.200035 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.199143    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f2jms"]
Sep 16 14:36:27.207569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.207542    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.210734 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.210691    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 14:36:27.210734 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.210707    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 14:36:27.210929 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.210803    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tdjzx\""
Sep 16 14:36:27.211066 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.211047    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 14:36:27.213514 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.213113    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b94269a4-3929-424a-bbb8-c50f384d1754-sys\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.213514 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.213160    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmwg\" (UniqueName: \"kubernetes.io/projected/b94269a4-3929-424a-bbb8-c50f384d1754-kube-api-access-dhmwg\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.213514 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.213235    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-accelerators-collector-config\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.213514 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.213267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-wtmp\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.213514 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.213313    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b94269a4-3929-424a-bbb8-c50f384d1754-root\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.213514 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.213339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.213514 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.213370    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-textfile\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.213514 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.213412    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-tls\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.213514 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.213436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94269a4-3929-424a-bbb8-c50f384d1754-metrics-client-ca\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.314450 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314423    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-accelerators-collector-config\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.314629 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314454    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-wtmp\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.314629 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314497    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b94269a4-3929-424a-bbb8-c50f384d1754-root\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.314629 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.314629 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-textfile\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.314629 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314582    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-tls\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.314629 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94269a4-3929-424a-bbb8-c50f384d1754-metrics-client-ca\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.314946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314632    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b94269a4-3929-424a-bbb8-c50f384d1754-sys\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.314946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314633    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-wtmp\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.314946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314665    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmwg\" (UniqueName: \"kubernetes.io/projected/b94269a4-3929-424a-bbb8-c50f384d1754-kube-api-access-dhmwg\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.314946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314908    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-textfile\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.315140 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.314963    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b94269a4-3929-424a-bbb8-c50f384d1754-root\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.315140 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.315081    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-accelerators-collector-config\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.315381 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.315343    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b94269a4-3929-424a-bbb8-c50f384d1754-sys\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.315588 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.315558    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b94269a4-3929-424a-bbb8-c50f384d1754-metrics-client-ca\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.318336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.318311    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.318426 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.318388    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b94269a4-3929-424a-bbb8-c50f384d1754-node-exporter-tls\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.326958 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.326937    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmwg\" (UniqueName: \"kubernetes.io/projected/b94269a4-3929-424a-bbb8-c50f384d1754-kube-api-access-dhmwg\") pod \"node-exporter-f2jms\" (UID: \"b94269a4-3929-424a-bbb8-c50f384d1754\") " pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.518185 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.518105    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f2jms"
Sep 16 14:36:27.538835 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:27.538804    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94269a4_3929_424a_bbb8_c50f384d1754.slice/crio-fbe6b28b74981eb28fea5e6b23204843c5f448658aae26588a272e1af5942c0e WatchSource:0}: Error finding container fbe6b28b74981eb28fea5e6b23204843c5f448658aae26588a272e1af5942c0e: Status 404 returned error can't find the container with id fbe6b28b74981eb28fea5e6b23204843c5f448658aae26588a272e1af5942c0e
Sep 16 14:36:27.603872 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.603847    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546568fdb5-cl296" event={"ID":"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146","Type":"ContainerStarted","Data":"d8ddb80562054684de89dd10bc536f01bb0c472f59ebe5e91eda186670923141"}
Sep 16 14:36:27.609006 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.608973    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f2jms" event={"ID":"b94269a4-3929-424a-bbb8-c50f384d1754","Type":"ContainerStarted","Data":"fbe6b28b74981eb28fea5e6b23204843c5f448658aae26588a272e1af5942c0e"}
Sep 16 14:36:27.627929 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:27.627876    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-546568fdb5-cl296" podStartSLOduration=0.737648113 podStartE2EDuration="3.62785617s" totalImagesPullingTime="2.890208057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:27.627392186 +0000 UTC m=+64.781470474" watchObservedRunningTime="2026-09-16 14:36:27.62785617 +0000 UTC m=+64.781934459"
Sep 16 14:36:28.021126 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.021085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:28.021336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.021205    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:28.029904 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.029880    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 14:36:28.030027 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.029963    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 14:36:28.034332 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.034314    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b52afd57-0df9-4a81-b6dc-a90212fcc659-metrics-certs\") pod \"network-metrics-daemon-qwq9w\" (UID: \"b52afd57-0df9-4a81-b6dc-a90212fcc659\") " pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:28.034434 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.034410    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/82cdd79b-96ab-48dd-a670-0d59194a7992-original-pull-secret\") pod \"global-pull-secret-syncer-r9l6v\" (UID: \"82cdd79b-96ab-48dd-a670-0d59194a7992\") " pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:28.121986 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.121951    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99ct\" (UniqueName: \"kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct\") pod \"network-check-target-n4fxc\" (UID: \"ecbbbd3b-6571-414d-94c1-130a385f4d1a\") " pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:28.127145 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.127124    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 14:36:28.142357 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.142336    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 14:36:28.145520 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.145501    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t99ct\" (UniqueName: \"kubernetes.io/projected/ecbbbd3b-6571-414d-94c1-130a385f4d1a-kube-api-access-t99ct\") pod \"network-check-target-n4fxc\" (UID: \"ecbbbd3b-6571-414d-94c1-130a385f4d1a\") " pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:28.300809 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.300738    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r9l6v"
Sep 16 14:36:28.310416 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.310393    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sr586\""
Sep 16 14:36:28.317546 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.317522    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:28.317928 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.317894    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bf8l5\""
Sep 16 14:36:28.321340 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.321323    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qwq9w"
Sep 16 14:36:28.448470 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.448431    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:36:28.453365 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.453338    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.459293 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.459092    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 14:36:28.459469 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.459359    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 14:36:28.459469 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.459373    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 14:36:28.459469 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.459428    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8sn2s\""
Sep 16 14:36:28.459469 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.459464    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 14:36:28.459622 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.459373    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 14:36:28.459622 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.459380    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 14:36:28.459622 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.459430    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 14:36:28.459760 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.459730    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 14:36:28.467025 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.467006    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 14:36:28.501654 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.501538    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524104    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524151    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524197    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524231    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-web-config\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524289    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-volume\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524318    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-out\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524348    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524484    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vsbn\" (UniqueName: \"kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-kube-api-access-4vsbn\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.524715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.524517    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.625344 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625320    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.625454 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625351    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.625454 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.625454 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625428    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-web-config\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.625614 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625465    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.625614 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625504    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-volume\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.625614 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625548    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-out\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.625614 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625580    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.626111 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625642    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.626111 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.626111 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625742    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.626111 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625767    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsbn\" (UniqueName: \"kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-kube-api-access-4vsbn\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.626111 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.625815    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.626111 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.626009    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.626429 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.626341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.626429 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:28.626361    2558 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 14:36:28.626429 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:36:28.626427    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-main-tls podName:7d95f3c9-aa4e-410b-8d55-9c6891ee5b17 nodeName:}" failed. No retries permitted until 2026-09-16 14:36:29.126409578 +0000 UTC m=+66.280487845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17") : secret "alertmanager-main-tls" not found
Sep 16 14:36:28.627517 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.627155    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.628312 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.628290    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-web-config\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.628617 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.628601    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.628770 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.628751    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-out\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.628846 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.628769    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.629104 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.629074    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.629408 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.629387    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-volume\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.630061 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.630011    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.630986 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.630956    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.640743 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.640722    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsbn\" (UniqueName: \"kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-kube-api-access-4vsbn\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:28.757751 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.757726    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qwq9w"]
Sep 16 14:36:28.762023 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:28.761987    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52afd57_0df9_4a81_b6dc_a90212fcc659.slice/crio-baddf1fc055f4bc9a3358dfe3e2542ca440a1ca77c88b14a159ac3c51c7ba900 WatchSource:0}: Error finding container baddf1fc055f4bc9a3358dfe3e2542ca440a1ca77c88b14a159ac3c51c7ba900: Status 404 returned error can't find the container with id baddf1fc055f4bc9a3358dfe3e2542ca440a1ca77c88b14a159ac3c51c7ba900
Sep 16 14:36:28.777943 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.777831    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n4fxc"]
Sep 16 14:36:28.780963 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:28.780941    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbbbd3b_6571_414d_94c1_130a385f4d1a.slice/crio-c166f4931eece0a8c14ec6211098b8fe02c72ae27ef0c192f5226f9cb5f22b63 WatchSource:0}: Error finding container c166f4931eece0a8c14ec6211098b8fe02c72ae27ef0c192f5226f9cb5f22b63: Status 404 returned error can't find the container with id c166f4931eece0a8c14ec6211098b8fe02c72ae27ef0c192f5226f9cb5f22b63
Sep 16 14:36:28.791233 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:28.791213    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r9l6v"]
Sep 16 14:36:28.794110 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:28.794088    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cdd79b_96ab_48dd_a670_0d59194a7992.slice/crio-3765078a20c2f8f663e6d41c8cea492fc7adee0a9bbf7a2b8a7ce61119d9e6fb WatchSource:0}: Error finding container 3765078a20c2f8f663e6d41c8cea492fc7adee0a9bbf7a2b8a7ce61119d9e6fb: Status 404 returned error can't find the container with id 3765078a20c2f8f663e6d41c8cea492fc7adee0a9bbf7a2b8a7ce61119d9e6fb
Sep 16 14:36:29.129469 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:29.129438    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:29.131779 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:29.131761    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:29.365180 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:29.365139    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:36:29.619689 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:29.619646    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r9l6v" event={"ID":"82cdd79b-96ab-48dd-a670-0d59194a7992","Type":"ContainerStarted","Data":"3765078a20c2f8f663e6d41c8cea492fc7adee0a9bbf7a2b8a7ce61119d9e6fb"}
Sep 16 14:36:29.620771 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:29.620746    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qwq9w" event={"ID":"b52afd57-0df9-4a81-b6dc-a90212fcc659","Type":"ContainerStarted","Data":"baddf1fc055f4bc9a3358dfe3e2542ca440a1ca77c88b14a159ac3c51c7ba900"}
Sep 16 14:36:29.621739 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:29.621715    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n4fxc" event={"ID":"ecbbbd3b-6571-414d-94c1-130a385f4d1a","Type":"ContainerStarted","Data":"c166f4931eece0a8c14ec6211098b8fe02c72ae27ef0c192f5226f9cb5f22b63"}
Sep 16 14:36:29.623387 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:29.623364    2558 generic.go:356] "Generic (PLEG): container finished" podID="b94269a4-3929-424a-bbb8-c50f384d1754" containerID="ade7bff12cb46af8851fe24815816689370dd8018698ba152340ae9ba102d667" exitCode=0
Sep 16 14:36:29.623477 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:29.623435    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f2jms" event={"ID":"b94269a4-3929-424a-bbb8-c50f384d1754","Type":"ContainerDied","Data":"ade7bff12cb46af8851fe24815816689370dd8018698ba152340ae9ba102d667"}
Sep 16 14:36:29.705128 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:29.705101    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:36:29.707788 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:29.707754    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d95f3c9_aa4e_410b_8d55_9c6891ee5b17.slice/crio-efa1b9969a5a8ec1fa9cf6e40ad69a71b4e30eec7498603cb910544e32553873 WatchSource:0}: Error finding container efa1b9969a5a8ec1fa9cf6e40ad69a71b4e30eec7498603cb910544e32553873: Status 404 returned error can't find the container with id efa1b9969a5a8ec1fa9cf6e40ad69a71b4e30eec7498603cb910544e32553873
Sep 16 14:36:30.576706 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:30.576681    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wqn5h"
Sep 16 14:36:30.633773 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:30.633739    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qwq9w" event={"ID":"b52afd57-0df9-4a81-b6dc-a90212fcc659","Type":"ContainerStarted","Data":"a15df0fb2d695011c89bbbb44709d34c5e2081dd302dc8412fc65b15c0f12a15"}
Sep 16 14:36:30.636871 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:30.636800    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerStarted","Data":"efa1b9969a5a8ec1fa9cf6e40ad69a71b4e30eec7498603cb910544e32553873"}
Sep 16 14:36:30.641198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:30.640933    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f2jms" event={"ID":"b94269a4-3929-424a-bbb8-c50f384d1754","Type":"ContainerStarted","Data":"7186800e8acdba803e3c064763edf4ad0a9f58595766f7ae2c37836666893485"}
Sep 16 14:36:30.641198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:30.640983    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f2jms" event={"ID":"b94269a4-3929-424a-bbb8-c50f384d1754","Type":"ContainerStarted","Data":"4b8020f3d209848b61b44428f5c2c6d00a16749e9caf47c2408ac939ec0bc1d7"}
Sep 16 14:36:31.240410 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.239881    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f2jms" podStartSLOduration=3.190779296 podStartE2EDuration="4.239855948s" totalImagesPullingTime="987.529407ms" totalInitContainerRuntime="61.547245ms" isStatefulPod=true podCreationTimestamp="2026-09-16 14:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:30.693602597 +0000 UTC m=+67.847680885" watchObservedRunningTime="2026-09-16 14:36:31.239855948 +0000 UTC m=+68.393934236"
Sep 16 14:36:31.240410 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.240268    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f4bdd5b6c-2nlrg"]
Sep 16 14:36:31.253570 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.253545    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.257498 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.257459    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f4bdd5b6c-2nlrg"]
Sep 16 14:36:31.264883 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.264860    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 14:36:31.346038 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.346008    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-trusted-ca-bundle\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.346153 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.346068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-service-ca\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.346153 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.346113    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-config\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.346271 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.346177    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-oauth-serving-cert\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.346271 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.346222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-serving-cert\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.346372 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.346284    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2p6\" (UniqueName: \"kubernetes.io/projected/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-kube-api-access-mq2p6\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.346372 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.346313    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-oauth-config\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.447567 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.447539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-oauth-config\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.447701 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.447574    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-trusted-ca-bundle\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.447701 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.447605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-service-ca\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.447701 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.447623    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-config\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.447701 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.447643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-oauth-serving-cert\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.447701 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.447671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-serving-cert\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.447701 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.447698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2p6\" (UniqueName: \"kubernetes.io/projected/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-kube-api-access-mq2p6\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.448347 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.448302    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-oauth-serving-cert\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.448447 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.448396    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-config\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.448447 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.448408    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-service-ca\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.448530 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.448467    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-trusted-ca-bundle\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.449962 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.449939    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-serving-cert\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.450043 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.449981    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-oauth-config\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.456862 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.456845    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2p6\" (UniqueName: \"kubernetes.io/projected/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-kube-api-access-mq2p6\") pod \"console-5f4bdd5b6c-2nlrg\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") " pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.564012 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.563979    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-684fbdf947-xtm97"]
Sep 16 14:36:31.566942 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.566922    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.567142 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.566982    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:31.570786 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.570760    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bbqw9\""
Sep 16 14:36:31.570890 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.570863    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 14:36:31.571077 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.571063    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7mlphi4ndhk2q\""
Sep 16 14:36:31.571242 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.571225    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 14:36:31.571317 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.571265    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 14:36:31.572549 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.572531    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 14:36:31.585620 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.585589    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-684fbdf947-xtm97"]
Sep 16 14:36:31.647123 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.647091    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qwq9w" event={"ID":"b52afd57-0df9-4a81-b6dc-a90212fcc659","Type":"ContainerStarted","Data":"fbdf0d01c67b8c2d9ef90bdd277f6168f4eb797e3854190c14638a32149c5944"}
Sep 16 14:36:31.648881 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.648848    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fcc44efc-a893-4e75-b805-765ab2aca444-secret-metrics-server-tls\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.649022 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.648884    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fcc44efc-a893-4e75-b805-765ab2aca444-secret-metrics-server-client-certs\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.649022 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.648924    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fcc44efc-a893-4e75-b805-765ab2aca444-audit-log\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.649022 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.648947    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fcc44efc-a893-4e75-b805-765ab2aca444-client-ca-bundle\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.649022 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.648966    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fcc44efc-a893-4e75-b805-765ab2aca444-metrics-server-audit-profiles\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.649022 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.648988    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9skx\" (UniqueName: \"kubernetes.io/projected/fcc44efc-a893-4e75-b805-765ab2aca444-kube-api-access-t9skx\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.649022 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.649017    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc44efc-a893-4e75-b805-765ab2aca444-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.649366 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.649282    2558 generic.go:356] "Generic (PLEG): container finished" podID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerID="ba4fefd86a6a2f47dfc41101aa4b7863c73f6a5cceb11e6f5c05fd4cc30d52b8" exitCode=0
Sep 16 14:36:31.649489 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.649468    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerDied","Data":"ba4fefd86a6a2f47dfc41101aa4b7863c73f6a5cceb11e6f5c05fd4cc30d52b8"}
Sep 16 14:36:31.670917 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.670860    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qwq9w" podStartSLOduration=67.070982826 podStartE2EDuration="1m8.670831974s" totalImagesPullingTime="1.599849148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:31.670027963 +0000 UTC m=+68.824106254" watchObservedRunningTime="2026-09-16 14:36:31.670831974 +0000 UTC m=+68.824910265"
Sep 16 14:36:31.749550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.749509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fcc44efc-a893-4e75-b805-765ab2aca444-audit-log\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.749756 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.749583    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fcc44efc-a893-4e75-b805-765ab2aca444-client-ca-bundle\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.749756 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.749638    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fcc44efc-a893-4e75-b805-765ab2aca444-metrics-server-audit-profiles\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.749756 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.749662    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9skx\" (UniqueName: \"kubernetes.io/projected/fcc44efc-a893-4e75-b805-765ab2aca444-kube-api-access-t9skx\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.749756 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.749696    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc44efc-a893-4e75-b805-765ab2aca444-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.749999 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.749964    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fcc44efc-a893-4e75-b805-765ab2aca444-secret-metrics-server-tls\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.750055 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.750002    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fcc44efc-a893-4e75-b805-765ab2aca444-secret-metrics-server-client-certs\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.750106 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.750049    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fcc44efc-a893-4e75-b805-765ab2aca444-audit-log\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.750572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.750548    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcc44efc-a893-4e75-b805-765ab2aca444-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.751442 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.751418    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fcc44efc-a893-4e75-b805-765ab2aca444-metrics-server-audit-profiles\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.753233 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.753208    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fcc44efc-a893-4e75-b805-765ab2aca444-client-ca-bundle\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.753320 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.753241    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fcc44efc-a893-4e75-b805-765ab2aca444-secret-metrics-server-client-certs\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.753320 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.753258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fcc44efc-a893-4e75-b805-765ab2aca444-secret-metrics-server-tls\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.763270 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.763246    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9skx\" (UniqueName: \"kubernetes.io/projected/fcc44efc-a893-4e75-b805-765ab2aca444-kube-api-access-t9skx\") pod \"metrics-server-684fbdf947-xtm97\" (UID: \"fcc44efc-a893-4e75-b805-765ab2aca444\") " pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:31.878272 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:31.877929    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:33.815303 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.815268    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:36:33.819237 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.819215    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.827267 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.827230    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 14:36:33.827418 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.827282    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 14:36:33.827418 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.827309    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 14:36:33.827418 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.827348    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 14:36:33.827418 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.827309    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 14:36:33.827628 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.827586    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 14:36:33.828186 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.828130    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 14:36:33.828901 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.828639    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 14:36:33.828901 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.828774    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-v6wvg\""
Sep 16 14:36:33.828901 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.828827    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 14:36:33.828901 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.828853    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 14:36:33.829600 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.829582    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 14:36:33.831286 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.831078    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2i78j051k51m2\""
Sep 16 14:36:33.831286 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.831181    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 14:36:33.834250 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.834230    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 14:36:33.838440 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.838420    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 14:36:33.855268 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.855237    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:36:33.869973 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.869937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870114 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870003    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870114 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870037    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870114 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870114 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870325 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z44rb\" (UniqueName: \"kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-kube-api-access-z44rb\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870325 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870151    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870325 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870192    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870325 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870219    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870325 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870325 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870325 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870300    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config-out\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870325    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870351    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870381    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870403    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-web-config\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870431    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870454    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870484    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.870569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.870509    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.970961 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.970922    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.970961 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.970959    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.970988    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971071    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971095    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971112    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971139    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44rb\" (UniqueName: \"kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-kube-api-access-z44rb\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971154    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971182    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971228    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971713 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971257    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971713 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config-out\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971713 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971713 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971713 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971374    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971713 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971397    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-web-config\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971713 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.971713 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971443    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.972090 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971731    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.972090 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971810    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.972090 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.971824    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.972498 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.972314    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.973902 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.973874    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.975524 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.974796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.975524 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.975004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.975524 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.975069    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.975728 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.975625    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.976054 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.975952    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.976468 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.976443    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config-out\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.976559 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.976496    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.976559 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.976508    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.976983 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.976957    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.977053 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.977002    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.977137 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.977117    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-web-config\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.977556 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.977499    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.977624 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.977583    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:33.978483 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:33.978433    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:34.002526 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.002501    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44rb\" (UniqueName: \"kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-kube-api-access-z44rb\") pod \"prometheus-k8s-0\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:34.137848 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.137782    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:34.372553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.372520    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:34.372693 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.372588    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:34.378786 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.378762    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:34.452262 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.452182    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f4bdd5b6c-2nlrg"]
Sep 16 14:36:34.456075 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:34.456045    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd03cdc_51c3_4d42_b0c5_2f02bccaed78.slice/crio-f2b22341b07772b317139fe1fad034121fe27d7238aed352e8a3e259d8b97b02 WatchSource:0}: Error finding container f2b22341b07772b317139fe1fad034121fe27d7238aed352e8a3e259d8b97b02: Status 404 returned error can't find the container with id f2b22341b07772b317139fe1fad034121fe27d7238aed352e8a3e259d8b97b02
Sep 16 14:36:34.530666 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.530508    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-684fbdf947-xtm97"]
Sep 16 14:36:34.533505 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:34.533463    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc44efc_a893_4e75_b805_765ab2aca444.slice/crio-7527755c7704ee13de85dc30b48ca2edb88dc46059744404943ff48bdfe0b38c WatchSource:0}: Error finding container 7527755c7704ee13de85dc30b48ca2edb88dc46059744404943ff48bdfe0b38c: Status 404 returned error can't find the container with id 7527755c7704ee13de85dc30b48ca2edb88dc46059744404943ff48bdfe0b38c
Sep 16 14:36:34.560131 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.560099    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:36:34.563432 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:36:34.563406    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4107a59_2d99_44f5_9b11_4ba2fe60a310.slice/crio-385889e55e0511d223458cf628f69c8402c10c2aff3f4cb6c512fd000fb06e31 WatchSource:0}: Error finding container 385889e55e0511d223458cf628f69c8402c10c2aff3f4cb6c512fd000fb06e31: Status 404 returned error can't find the container with id 385889e55e0511d223458cf628f69c8402c10c2aff3f4cb6c512fd000fb06e31
Sep 16 14:36:34.666256 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.666220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r9l6v" event={"ID":"82cdd79b-96ab-48dd-a670-0d59194a7992","Type":"ContainerStarted","Data":"55033999b7be2789cb12ddf54924d1ed34c385e1d478b3ff44303e1079913db5"}
Sep 16 14:36:34.668792 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.668355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n4fxc" event={"ID":"ecbbbd3b-6571-414d-94c1-130a385f4d1a","Type":"ContainerStarted","Data":"42599fcb74add05463aeb1fde0c25b35884e6faade9929b86c103924eece5a78"}
Sep 16 14:36:34.668792 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.668745    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:36:34.671033 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.670965    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4bdd5b6c-2nlrg" event={"ID":"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78","Type":"ContainerStarted","Data":"5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282"}
Sep 16 14:36:34.671033 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.671006    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4bdd5b6c-2nlrg" event={"ID":"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78","Type":"ContainerStarted","Data":"f2b22341b07772b317139fe1fad034121fe27d7238aed352e8a3e259d8b97b02"}
Sep 16 14:36:34.673061 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.673033    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-684fbdf947-xtm97" event={"ID":"fcc44efc-a893-4e75-b805-765ab2aca444","Type":"ContainerStarted","Data":"7527755c7704ee13de85dc30b48ca2edb88dc46059744404943ff48bdfe0b38c"}
Sep 16 14:36:34.675050 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.675022    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerStarted","Data":"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"}
Sep 16 14:36:34.675148 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.675095    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerStarted","Data":"385889e55e0511d223458cf628f69c8402c10c2aff3f4cb6c512fd000fb06e31"}
Sep 16 14:36:34.679890 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.679868    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:36:34.704511 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.704416    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r9l6v" podStartSLOduration=66.280273882 podStartE2EDuration="1m11.704397469s" totalImagesPullingTime="5.424123587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:34.704287486 +0000 UTC m=+71.858365772" watchObservedRunningTime="2026-09-16 14:36:34.704397469 +0000 UTC m=+71.858475760"
Sep 16 14:36:34.743746 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.743700    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n4fxc" podStartSLOduration=66.314294204 podStartE2EDuration="1m11.743684728s" totalImagesPullingTime="5.429390524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:34.742949715 +0000 UTC m=+71.897028103" watchObservedRunningTime="2026-09-16 14:36:34.743684728 +0000 UTC m=+71.897763016"
Sep 16 14:36:34.927746 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:34.927688    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f4bdd5b6c-2nlrg" podStartSLOduration=3.927659149 podStartE2EDuration="3.927659149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:34.922295222 +0000 UTC m=+72.076373509" watchObservedRunningTime="2026-09-16 14:36:34.927659149 +0000 UTC m=+72.081737437"
Sep 16 14:36:35.681073 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:35.680992    2558 generic.go:356] "Generic (PLEG): container finished" podID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerID="dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217" exitCode=0
Sep 16 14:36:35.681239 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:35.681089    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerDied","Data":"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"}
Sep 16 14:36:35.684989 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:35.684961    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerStarted","Data":"1f9e19d0af5800e8df2b7b92962d2e91c9ea2588afab8543efcd1bf142f34bc9"}
Sep 16 14:36:35.685096 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:35.685047    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerStarted","Data":"9975a96b8ad35e9ce0a4c98efd074224bda1588b768d48fa1869fec955c51592"}
Sep 16 14:36:35.685096 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:35.685085    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerStarted","Data":"c94cd128b268307763cbb2fd38134064eecae66220a424726330b9279ec0f881"}
Sep 16 14:36:35.685207 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:35.685111    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerStarted","Data":"1c5e8f8f8973d309006d0c1a6d7c757f9d4f4a51d7be91e8f2a5c9a42fca9cb9"}
Sep 16 14:36:35.685207 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:35.685134    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerStarted","Data":"ac45cc46fbef0f2b47062f6194e2a27a57faec4028e0c09f99537d57797d9560"}
Sep 16 14:36:36.155747 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:36.155719    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f4bdd5b6c-2nlrg"]
Sep 16 14:36:36.690077 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:36.690035    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-684fbdf947-xtm97" event={"ID":"fcc44efc-a893-4e75-b805-765ab2aca444","Type":"ContainerStarted","Data":"444f6bebc7bb2b9ac697ad0dce7738ff1f3cb7c9e782fafc430df043fb920205"}
Sep 16 14:36:36.693372 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:36.693340    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerStarted","Data":"b5d3da965abea25858df61c105bd02ef496107cf11cbb085dd2b1afbb8149228"}
Sep 16 14:36:36.730626 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:36.730525    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-684fbdf947-xtm97" podStartSLOduration=4.156425 podStartE2EDuration="5.730508825s" totalImagesPullingTime="1.574083825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:36.729999262 +0000 UTC m=+73.884077587" watchObservedRunningTime="2026-09-16 14:36:36.730508825 +0000 UTC m=+73.884587112"
Sep 16 14:36:36.785533 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:36.785481    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.654805551 podStartE2EDuration="8.785463943s" totalImagesPullingTime="6.111354555s" totalInitContainerRuntime="19.303837ms" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:36.784373637 +0000 UTC m=+73.938451924" watchObservedRunningTime="2026-09-16 14:36:36.785463943 +0000 UTC m=+73.939542230"
Sep 16 14:36:40.713971 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:40.713933    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerStarted","Data":"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"}
Sep 16 14:36:40.714414 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:40.714004    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerStarted","Data":"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"}
Sep 16 14:36:41.568086 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:41.568039    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:36:42.726297 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:42.726257    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerStarted","Data":"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"}
Sep 16 14:36:42.726730 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:42.726318    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerStarted","Data":"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"}
Sep 16 14:36:42.726730 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:42.726345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerStarted","Data":"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"}
Sep 16 14:36:42.726730 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:42.726370    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerStarted","Data":"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"}
Sep 16 14:36:42.770873 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:42.770809    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.552478994 podStartE2EDuration="9.770786344s" totalImagesPullingTime="6.194991669s" totalInitContainerRuntime="23.315681ms" isStatefulPod=false podCreationTimestamp="2026-09-16 14:36:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:36:42.767463223 +0000 UTC m=+79.921541509" watchObservedRunningTime="2026-09-16 14:36:42.770786344 +0000 UTC m=+79.924864633"
Sep 16 14:36:44.138111 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:44.138076    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:36:47.593489 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:47.593362    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546568fdb5-cl296"]
Sep 16 14:36:51.878416 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:51.878381    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:36:51.878766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:36:51.878446    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:37:01.711652 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:01.711614    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f4bdd5b6c-2nlrg" podUID="dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" containerName="console" containerID="cri-o://5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282" gracePeriod=15
Sep 16 14:37:01.976413 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:01.976391    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:37:02.130181 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.130146    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-service-ca\") pod \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") "
Sep 16 14:37:02.130347 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.130215    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-serving-cert\") pod \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") "
Sep 16 14:37:02.130347 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.130258    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-trusted-ca-bundle\") pod \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") "
Sep 16 14:37:02.130347 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.130291    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq2p6\" (UniqueName: \"kubernetes.io/projected/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-kube-api-access-mq2p6\") pod \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") "
Sep 16 14:37:02.130539 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.130511    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-service-ca" (OuterVolumeSpecName: "service-ca") pod "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" (UID: "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:02.130710 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.130689    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" (UID: "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:02.130921 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.130902    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-oauth-config\") pod \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") "
Sep 16 14:37:02.131000 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.130986    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-config\") pod \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") "
Sep 16 14:37:02.131056 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.131036    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-oauth-serving-cert\") pod \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\" (UID: \"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78\") "
Sep 16 14:37:02.131713 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.131674    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-config" (OuterVolumeSpecName: "console-config") pod "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" (UID: "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:02.131851 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.131736    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:02.131851 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.131768    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-trusted-ca-bundle\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:02.132861 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.132763    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" (UID: "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:02.134212 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.133212    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-kube-api-access-mq2p6" (OuterVolumeSpecName: "kube-api-access-mq2p6") pod "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" (UID: "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78"). InnerVolumeSpecName "kube-api-access-mq2p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:37:02.136244 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.136215    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" (UID: "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:02.137073 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.137045    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" (UID: "dcd03cdc-51c3-4d42-b0c5-2f02bccaed78"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:02.232415 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.232330    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq2p6\" (UniqueName: \"kubernetes.io/projected/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-kube-api-access-mq2p6\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:02.232415 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.232362    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-oauth-config\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:02.232415 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.232372    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-config\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:02.232415 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.232381    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-oauth-serving-cert\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:02.232415 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.232390    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78-console-serving-cert\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:02.801887 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.801852    2558 generic.go:356] "Generic (PLEG): container finished" podID="dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" containerID="5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282" exitCode=0
Sep 16 14:37:02.802318 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.801918    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f4bdd5b6c-2nlrg"
Sep 16 14:37:02.802318 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.801939    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4bdd5b6c-2nlrg" event={"ID":"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78","Type":"ContainerDied","Data":"5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282"}
Sep 16 14:37:02.802318 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.801999    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f4bdd5b6c-2nlrg" event={"ID":"dcd03cdc-51c3-4d42-b0c5-2f02bccaed78","Type":"ContainerDied","Data":"f2b22341b07772b317139fe1fad034121fe27d7238aed352e8a3e259d8b97b02"}
Sep 16 14:37:02.802318 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.802026    2558 scope.go:117] "RemoveContainer" containerID="5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282"
Sep 16 14:37:02.810728 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.810707    2558 scope.go:117] "RemoveContainer" containerID="5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282"
Sep 16 14:37:02.811049 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:37:02.811027    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282\": container with ID starting with 5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282 not found: ID does not exist" containerID="5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282"
Sep 16 14:37:02.811101 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.811060    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282"} err="failed to get container status \"5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282\": rpc error: code = NotFound desc = could not find container \"5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282\": container with ID starting with 5f07c600c60efcdd281e501180bdfc3c178fb0a522db0f788a08dcb68d73d282 not found: ID does not exist"
Sep 16 14:37:02.823740 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.823714    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f4bdd5b6c-2nlrg"]
Sep 16 14:37:02.829495 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:02.829472    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f4bdd5b6c-2nlrg"]
Sep 16 14:37:03.294402 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:03.294372    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" path="/var/lib/kubelet/pods/dcd03cdc-51c3-4d42-b0c5-2f02bccaed78/volumes"
Sep 16 14:37:06.695695 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:06.695664    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n4fxc"
Sep 16 14:37:11.883489 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:11.883459    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:37:11.887401 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:11.887379    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-684fbdf947-xtm97"
Sep 16 14:37:12.612462 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.612425    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-546568fdb5-cl296" podUID="79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" containerName="console" containerID="cri-o://d8ddb80562054684de89dd10bc536f01bb0c472f59ebe5e91eda186670923141" gracePeriod=15
Sep 16 14:37:12.841678 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.841647    2558 generic.go:356] "Generic (PLEG): container finished" podID="79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" containerID="d8ddb80562054684de89dd10bc536f01bb0c472f59ebe5e91eda186670923141" exitCode=0
Sep 16 14:37:12.841837 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.841721    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546568fdb5-cl296" event={"ID":"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146","Type":"ContainerDied","Data":"d8ddb80562054684de89dd10bc536f01bb0c472f59ebe5e91eda186670923141"}
Sep 16 14:37:12.870375 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.870318    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:37:12.914543 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.914515    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-oauth-config\") pod \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") "
Sep 16 14:37:12.914991 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.914586    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd4vf\" (UniqueName: \"kubernetes.io/projected/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-kube-api-access-rd4vf\") pod \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") "
Sep 16 14:37:12.914991 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.914625    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-serving-cert\") pod \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") "
Sep 16 14:37:12.914991 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.914668    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-config\") pod \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") "
Sep 16 14:37:12.914991 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.914709    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-service-ca\") pod \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") "
Sep 16 14:37:12.914991 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.914753    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-oauth-serving-cert\") pod \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\" (UID: \"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146\") "
Sep 16 14:37:12.915279 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.915091    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-config" (OuterVolumeSpecName: "console-config") pod "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" (UID: "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:12.915279 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.915195    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-service-ca" (OuterVolumeSpecName: "service-ca") pod "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" (UID: "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:12.915599 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.915577    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" (UID: "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:12.917174 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.917118    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" (UID: "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:12.917336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.917309    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-kube-api-access-rd4vf" (OuterVolumeSpecName: "kube-api-access-rd4vf") pod "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" (UID: "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146"). InnerVolumeSpecName "kube-api-access-rd4vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:37:12.917412 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:12.917386    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" (UID: "79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:13.015512 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:13.015474    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-serving-cert\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:13.015512 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:13.015507    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-config\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:13.015512 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:13.015516    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:13.015512 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:13.015524    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-oauth-serving-cert\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:13.015766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:13.015533    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-console-oauth-config\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:13.015766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:13.015541    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd4vf\" (UniqueName: \"kubernetes.io/projected/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146-kube-api-access-rd4vf\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:13.846699 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:13.846663    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-546568fdb5-cl296" event={"ID":"79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146","Type":"ContainerDied","Data":"4413e478d188f3bdcc8785e30020d24eb246bb2a89d9bb2e117973a1f688833d"}
Sep 16 14:37:13.846699 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:13.846689    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-546568fdb5-cl296"
Sep 16 14:37:13.846917 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:13.846722    2558 scope.go:117] "RemoveContainer" containerID="d8ddb80562054684de89dd10bc536f01bb0c472f59ebe5e91eda186670923141"
Sep 16 14:37:13.862886 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:13.862862    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-546568fdb5-cl296"]
Sep 16 14:37:13.869035 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:13.869014    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-546568fdb5-cl296"]
Sep 16 14:37:15.293757 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:15.293729    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" path="/var/lib/kubelet/pods/79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146/volumes"
Sep 16 14:37:34.138995 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:34.138951    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:34.157046 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:34.157019    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:34.937727 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:34.937702    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:47.772479 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.772441    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:37:47.773097 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.772760    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="alertmanager" containerID="cri-o://ac45cc46fbef0f2b47062f6194e2a27a57faec4028e0c09f99537d57797d9560" gracePeriod=120
Sep 16 14:37:47.773097 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.772817    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy-metric" containerID="cri-o://1f9e19d0af5800e8df2b7b92962d2e91c9ea2588afab8543efcd1bf142f34bc9" gracePeriod=120
Sep 16 14:37:47.773097 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.772860    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy-web" containerID="cri-o://c94cd128b268307763cbb2fd38134064eecae66220a424726330b9279ec0f881" gracePeriod=120
Sep 16 14:37:47.773097 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.772892    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="config-reloader" containerID="cri-o://1c5e8f8f8973d309006d0c1a6d7c757f9d4f4a51d7be91e8f2a5c9a42fca9cb9" gracePeriod=120
Sep 16 14:37:47.773097 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.772860    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy" containerID="cri-o://9975a96b8ad35e9ce0a4c98efd074224bda1588b768d48fa1869fec955c51592" gracePeriod=120
Sep 16 14:37:47.773436 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.773300    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="prom-label-proxy" containerID="cri-o://b5d3da965abea25858df61c105bd02ef496107cf11cbb085dd2b1afbb8149228" gracePeriod=120
Sep 16 14:37:47.973657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.973626    2558 generic.go:356] "Generic (PLEG): container finished" podID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerID="b5d3da965abea25858df61c105bd02ef496107cf11cbb085dd2b1afbb8149228" exitCode=0
Sep 16 14:37:47.973657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.973650    2558 generic.go:356] "Generic (PLEG): container finished" podID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerID="9975a96b8ad35e9ce0a4c98efd074224bda1588b768d48fa1869fec955c51592" exitCode=0
Sep 16 14:37:47.973657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.973656    2558 generic.go:356] "Generic (PLEG): container finished" podID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerID="1c5e8f8f8973d309006d0c1a6d7c757f9d4f4a51d7be91e8f2a5c9a42fca9cb9" exitCode=0
Sep 16 14:37:47.973657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.973662    2558 generic.go:356] "Generic (PLEG): container finished" podID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerID="ac45cc46fbef0f2b47062f6194e2a27a57faec4028e0c09f99537d57797d9560" exitCode=0
Sep 16 14:37:47.973899 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.973683    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerDied","Data":"b5d3da965abea25858df61c105bd02ef496107cf11cbb085dd2b1afbb8149228"}
Sep 16 14:37:47.973899 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.973768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerDied","Data":"9975a96b8ad35e9ce0a4c98efd074224bda1588b768d48fa1869fec955c51592"}
Sep 16 14:37:47.973899 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.973798    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerDied","Data":"1c5e8f8f8973d309006d0c1a6d7c757f9d4f4a51d7be91e8f2a5c9a42fca9cb9"}
Sep 16 14:37:47.973899 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:47.973824    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerDied","Data":"ac45cc46fbef0f2b47062f6194e2a27a57faec4028e0c09f99537d57797d9560"}
Sep 16 14:37:48.980915 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:48.980887    2558 generic.go:356] "Generic (PLEG): container finished" podID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerID="1f9e19d0af5800e8df2b7b92962d2e91c9ea2588afab8543efcd1bf142f34bc9" exitCode=0
Sep 16 14:37:48.980915 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:48.980908    2558 generic.go:356] "Generic (PLEG): container finished" podID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerID="c94cd128b268307763cbb2fd38134064eecae66220a424726330b9279ec0f881" exitCode=0
Sep 16 14:37:48.981284 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:48.980955    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerDied","Data":"1f9e19d0af5800e8df2b7b92962d2e91c9ea2588afab8543efcd1bf142f34bc9"}
Sep 16 14:37:48.981284 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:48.981014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerDied","Data":"c94cd128b268307763cbb2fd38134064eecae66220a424726330b9279ec0f881"}
Sep 16 14:37:49.014363 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.014342    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:49.099979 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.099896    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-metrics-client-ca\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.099979 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.099938    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-web\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100199 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.099978    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-cluster-tls-config\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100199 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100013    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-out\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100199 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100040    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-volume\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100199 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100072    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-main-db\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100199 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100095    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vsbn\" (UniqueName: \"kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-kube-api-access-4vsbn\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100199 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100130    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100199 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100179    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100521 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100207    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-trusted-ca-bundle\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100521 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100233    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-tls-assets\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100521 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100268    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-main-tls\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100521 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100308    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-web-config\") pod \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\" (UID: \"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17\") "
Sep 16 14:37:49.100521 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100410    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:37:49.100754 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100564    2558 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-main-db\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.101143 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100910    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:49.101143 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.100946    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:49.103981 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.103954    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-out" (OuterVolumeSpecName: "config-out") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:37:49.104375 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.104318    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:49.104375 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.104337    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-kube-api-access-4vsbn" (OuterVolumeSpecName: "kube-api-access-4vsbn") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "kube-api-access-4vsbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:37:49.104552 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.104497    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-volume" (OuterVolumeSpecName: "config-volume") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:49.104614 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.104563    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:37:49.104666 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.104634    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:49.104791 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.104775    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:49.104879 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.104858    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:49.108410 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.108389    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:49.114365 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.114343    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-web-config" (OuterVolumeSpecName: "web-config") pod "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" (UID: "7d95f3c9-aa4e-410b-8d55-9c6891ee5b17"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:49.201410 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201378    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vsbn\" (UniqueName: \"kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-kube-api-access-4vsbn\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.201410 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201403    2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.201592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201418    2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.201592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201432    2558 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.201592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201446    2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-tls-assets\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.201592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201457    2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-main-tls\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.201592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201469    2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-web-config\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.201592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201480    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-metrics-client-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.201592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201492    2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.201592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201504    2558 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-cluster-tls-config\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.201592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201516    2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-out\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.201592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.201527    2558 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:49.987804 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.987771    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7d95f3c9-aa4e-410b-8d55-9c6891ee5b17","Type":"ContainerDied","Data":"efa1b9969a5a8ec1fa9cf6e40ad69a71b4e30eec7498603cb910544e32553873"}
Sep 16 14:37:49.988216 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.987818    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:49.988216 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.987858    2558 scope.go:117] "RemoveContainer" containerID="b5d3da965abea25858df61c105bd02ef496107cf11cbb085dd2b1afbb8149228"
Sep 16 14:37:49.995588 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:49.995571    2558 scope.go:117] "RemoveContainer" containerID="1f9e19d0af5800e8df2b7b92962d2e91c9ea2588afab8543efcd1bf142f34bc9"
Sep 16 14:37:50.001942 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.001926    2558 scope.go:117] "RemoveContainer" containerID="9975a96b8ad35e9ce0a4c98efd074224bda1588b768d48fa1869fec955c51592"
Sep 16 14:37:50.008375 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.008358    2558 scope.go:117] "RemoveContainer" containerID="c94cd128b268307763cbb2fd38134064eecae66220a424726330b9279ec0f881"
Sep 16 14:37:50.010782 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.010762    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:37:50.015583 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.015563    2558 scope.go:117] "RemoveContainer" containerID="1c5e8f8f8973d309006d0c1a6d7c757f9d4f4a51d7be91e8f2a5c9a42fca9cb9"
Sep 16 14:37:50.015642 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.015620    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:37:50.022072 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.022058    2558 scope.go:117] "RemoveContainer" containerID="ac45cc46fbef0f2b47062f6194e2a27a57faec4028e0c09f99537d57797d9560"
Sep 16 14:37:50.028575 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.028561    2558 scope.go:117] "RemoveContainer" containerID="ba4fefd86a6a2f47dfc41101aa4b7863c73f6a5cceb11e6f5c05fd4cc30d52b8"
Sep 16 14:37:50.045098 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045078    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:37:50.045330 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045318    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" containerName="console"
Sep 16 14:37:50.045374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045332    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" containerName="console"
Sep 16 14:37:50.045374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045340    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="config-reloader"
Sep 16 14:37:50.045374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045346    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="config-reloader"
Sep 16 14:37:50.045374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045355    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy-metric"
Sep 16 14:37:50.045374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045361    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy-metric"
Sep 16 14:37:50.045374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045368    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="alertmanager"
Sep 16 14:37:50.045374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045373    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="alertmanager"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045381    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="init-config-reloader"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045388    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="init-config-reloader"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045395    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045401    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045407    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy-web"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045412    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy-web"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045417    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="prom-label-proxy"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045422    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="prom-label-proxy"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045435    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" containerName="console"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045440    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" containerName="console"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045475    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="config-reloader"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045481    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045488    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="prom-label-proxy"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045493    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="alertmanager"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045498    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy-metric"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045502    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd03cdc-51c3-4d42-b0c5-2f02bccaed78" containerName="console"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045507    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" containerName="kube-rbac-proxy-web"
Sep 16 14:37:50.045572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.045512    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="79f5bc1e-e8b1-49d3-acf2-4a8e9dd33146" containerName="console"
Sep 16 14:37:50.050244 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.050227    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.053336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.053315    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8sn2s\""
Sep 16 14:37:50.053336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.053332    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 14:37:50.053489 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.053321    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 14:37:50.053489 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.053406    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 14:37:50.053667 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.053652    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 14:37:50.053797 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.053782    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 14:37:50.053866 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.053831    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 14:37:50.053866 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.053843    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 14:37:50.054337 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.054324    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 14:37:50.063245 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.063227    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 14:37:50.072497 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.072479    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:37:50.208820 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.208712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.208820 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.208763    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-config-volume\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.208820 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.208789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.208820 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.208819    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.209115 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.208839    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-web-config\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.209115 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.208925    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.209115 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.208951    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.209115 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.208991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe6edb56-54eb-4e31-a385-98ea97e6feb6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.209115 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.209050    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe6edb56-54eb-4e31-a385-98ea97e6feb6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.209115 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.209068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe6edb56-54eb-4e31-a385-98ea97e6feb6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.209115 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.209084    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe6edb56-54eb-4e31-a385-98ea97e6feb6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.209115 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.209110    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe6edb56-54eb-4e31-a385-98ea97e6feb6-config-out\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.209403 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.209142    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndcm\" (UniqueName: \"kubernetes.io/projected/fe6edb56-54eb-4e31-a385-98ea97e6feb6-kube-api-access-2ndcm\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310413 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe6edb56-54eb-4e31-a385-98ea97e6feb6-config-out\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310413 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndcm\" (UniqueName: \"kubernetes.io/projected/fe6edb56-54eb-4e31-a385-98ea97e6feb6-kube-api-access-2ndcm\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310591 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310591 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310458    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-config-volume\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310591 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310574    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310718 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310614    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310718 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-web-config\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310718 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310678    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310718 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310703    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310925 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe6edb56-54eb-4e31-a385-98ea97e6feb6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310925 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe6edb56-54eb-4e31-a385-98ea97e6feb6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310925 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310817    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe6edb56-54eb-4e31-a385-98ea97e6feb6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.310925 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.310846    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe6edb56-54eb-4e31-a385-98ea97e6feb6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.311609 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.311577    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe6edb56-54eb-4e31-a385-98ea97e6feb6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.313317 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.313291    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe6edb56-54eb-4e31-a385-98ea97e6feb6-config-out\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.313596 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.313571    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-config-volume\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.313670 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.313643    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.313670 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.313661    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fe6edb56-54eb-4e31-a385-98ea97e6feb6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.313811 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.313788    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.314003 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.313979    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-web-config\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.314153 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.314123    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.314337 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.314318    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe6edb56-54eb-4e31-a385-98ea97e6feb6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.314430 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.314410    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.314494 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.314473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fe6edb56-54eb-4e31-a385-98ea97e6feb6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.315521 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.315505    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe6edb56-54eb-4e31-a385-98ea97e6feb6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.320214 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.320195    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndcm\" (UniqueName: \"kubernetes.io/projected/fe6edb56-54eb-4e31-a385-98ea97e6feb6-kube-api-access-2ndcm\") pod \"alertmanager-main-0\" (UID: \"fe6edb56-54eb-4e31-a385-98ea97e6feb6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.359219 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.359200    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 14:37:50.645272 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.645223    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 14:37:50.993915 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.993824    2558 generic.go:356] "Generic (PLEG): container finished" podID="fe6edb56-54eb-4e31-a385-98ea97e6feb6" containerID="db7bf1b3f61af95515517cbefd6189ec6ecf0cba00149a9d4b5f2b9b4c8b2a66" exitCode=0
Sep 16 14:37:50.993915 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.993876    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe6edb56-54eb-4e31-a385-98ea97e6feb6","Type":"ContainerDied","Data":"db7bf1b3f61af95515517cbefd6189ec6ecf0cba00149a9d4b5f2b9b4c8b2a66"}
Sep 16 14:37:50.994437 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:50.993928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe6edb56-54eb-4e31-a385-98ea97e6feb6","Type":"ContainerStarted","Data":"01ebe240e91d4d04b34cc8a33f23580b23be5a66d812c458ee9ed887cc2994e7"}
Sep 16 14:37:51.297851 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:51.297819    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d95f3c9-aa4e-410b-8d55-9c6891ee5b17" path="/var/lib/kubelet/pods/7d95f3c9-aa4e-410b-8d55-9c6891ee5b17/volumes"
Sep 16 14:37:52.002990 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.002943    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe6edb56-54eb-4e31-a385-98ea97e6feb6","Type":"ContainerStarted","Data":"5c01e0b7d6853329ee078623d51870b0e653eef5f61dc75207da61b242964960"}
Sep 16 14:37:52.003494 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.003040    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe6edb56-54eb-4e31-a385-98ea97e6feb6","Type":"ContainerStarted","Data":"f320be5d1da10003509ee5ca61f44cb639c8bed5f2399d2f24b9db1359e6624e"}
Sep 16 14:37:52.003494 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.003104    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe6edb56-54eb-4e31-a385-98ea97e6feb6","Type":"ContainerStarted","Data":"dce3ceb68b1353c770af92085106f410c532173bb2d2fd0655db4b4dee78a746"}
Sep 16 14:37:52.003494 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.003146    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe6edb56-54eb-4e31-a385-98ea97e6feb6","Type":"ContainerStarted","Data":"ccffbf43105fbf16fa4f9fd8e5d3cf6a1e5e784371fdaac833d0ce58dfa8d9d6"}
Sep 16 14:37:52.003494 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.003205    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe6edb56-54eb-4e31-a385-98ea97e6feb6","Type":"ContainerStarted","Data":"51bd21d8283fe519218adfd12148a4881eecbf340fee6afff47408d74df09d80"}
Sep 16 14:37:52.003494 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.003244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fe6edb56-54eb-4e31-a385-98ea97e6feb6","Type":"ContainerStarted","Data":"5428fe6c063595c7dde21091a82bcd710f62bef4237361db53c8fe1669827947"}
Sep 16 14:37:52.034734 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.034669    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.016434583 podStartE2EDuration="2.034645726s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.211143ms" isStatefulPod=false podCreationTimestamp="2026-09-16 14:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:37:52.033250911 +0000 UTC m=+149.187329199" watchObservedRunningTime="2026-09-16 14:37:52.034645726 +0000 UTC m=+149.188724016"
Sep 16 14:37:52.117505 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.117468    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:37:52.117827 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.117796    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="prometheus" containerID="cri-o://3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983" gracePeriod=600
Sep 16 14:37:52.117956 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.117839    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="config-reloader" containerID="cri-o://11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5" gracePeriod=600
Sep 16 14:37:52.117956 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.117885    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy" containerID="cri-o://f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a" gracePeriod=600
Sep 16 14:37:52.117956 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.117904    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca" gracePeriod=600
Sep 16 14:37:52.118097 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.117983    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy-web" containerID="cri-o://0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072" gracePeriod=600
Sep 16 14:37:52.118143 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.118094    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="thanos-sidecar" containerID="cri-o://5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e" gracePeriod=600
Sep 16 14:37:52.364134 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.364103    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:52.532427 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532397    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config-out\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.532599 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532447    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-kubelet-serving-ca-bundle\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.532599 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532473    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-2\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.532599 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532502    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-trusted-ca-bundle\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.532599 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532539    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.532599 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532572    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-0\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.532857 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532688    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-grpc-tls\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.532857 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532730    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-db\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.532857 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532774    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-thanos-prometheus-http-client-file\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.532857 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532790    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:52.532857 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532802    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-tls-assets\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.532857 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532847    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.533116 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532877    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-1\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.533116 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532893    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-tls\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.533116 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532926    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.533116 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532965    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-web-config\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.533116 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.532992    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-metrics-client-ca\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.533116 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.533016    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-metrics-client-certs\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.533116 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.533042    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-kube-rbac-proxy\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.533116 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.533065    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-serving-certs-ca-bundle\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.534697 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.534651    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:37:52.534856 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.534694    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z44rb\" (UniqueName: \"kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-kube-api-access-z44rb\") pod \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\" (UID: \"a4107a59-2d99-44f5-9b11-4ba2fe60a310\") "
Sep 16 14:37:52.535961 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.535246    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:52.536107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.535557    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:52.536336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.536124    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-db\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.536336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.536245    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.536336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.535576    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:52.536336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.535674    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:52.536336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.535682    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:52.536336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.535855    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:52.536336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.535876    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:52.536336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.535974    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:37:52.536336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.536022    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config-out" (OuterVolumeSpecName: "config-out") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:37:52.536826 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.536372    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:37:52.536826 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.536460    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config" (OuterVolumeSpecName: "config") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:52.536826 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.536687    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:52.537367 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.537338    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:52.537533 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.537509    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:52.537947 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.537927    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-kube-api-access-z44rb" (OuterVolumeSpecName: "kube-api-access-z44rb") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "kube-api-access-z44rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:37:52.538403 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.538383    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:52.538482 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.538459    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:52.545734 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.545692    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-web-config" (OuterVolumeSpecName: "web-config") pod "a4107a59-2d99-44f5-9b11-4ba2fe60a310" (UID: "a4107a59-2d99-44f5-9b11-4ba2fe60a310"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:37:52.637452 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637422    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637452 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637449    2558 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-grpc-tls\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637452 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637461    2558 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637470    2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-tls-assets\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637480    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637490    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637500    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637508    2558 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637515    2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-web-config\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637523    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-metrics-client-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637530    2558 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-metrics-client-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637538    2558 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-kube-rbac-proxy\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637546    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637555    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z44rb\" (UniqueName: \"kubernetes.io/projected/a4107a59-2d99-44f5-9b11-4ba2fe60a310-kube-api-access-z44rb\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637563    2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a4107a59-2d99-44f5-9b11-4ba2fe60a310-config-out\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637570    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637578    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4107a59-2d99-44f5-9b11-4ba2fe60a310-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:52.637657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:52.637586    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a4107a59-2d99-44f5-9b11-4ba2fe60a310-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:37:53.010374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010342    2558 generic.go:356] "Generic (PLEG): container finished" podID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerID="ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca" exitCode=0
Sep 16 14:37:53.010374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010366    2558 generic.go:356] "Generic (PLEG): container finished" podID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerID="f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a" exitCode=0
Sep 16 14:37:53.010374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010372    2558 generic.go:356] "Generic (PLEG): container finished" podID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerID="0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072" exitCode=0
Sep 16 14:37:53.010374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010377    2558 generic.go:356] "Generic (PLEG): container finished" podID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerID="5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e" exitCode=0
Sep 16 14:37:53.010374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010383    2558 generic.go:356] "Generic (PLEG): container finished" podID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerID="11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5" exitCode=0
Sep 16 14:37:53.010962 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010388    2558 generic.go:356] "Generic (PLEG): container finished" podID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerID="3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983" exitCode=0
Sep 16 14:37:53.010962 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010412    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.010962 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010425    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerDied","Data":"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"}
Sep 16 14:37:53.010962 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010516    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerDied","Data":"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"}
Sep 16 14:37:53.010962 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010545    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerDied","Data":"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"}
Sep 16 14:37:53.010962 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010574    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerDied","Data":"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"}
Sep 16 14:37:53.010962 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010587    2558 scope.go:117] "RemoveContainer" containerID="ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"
Sep 16 14:37:53.010962 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010598    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerDied","Data":"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"}
Sep 16 14:37:53.010962 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010640    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerDied","Data":"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"}
Sep 16 14:37:53.010962 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.010664    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a4107a59-2d99-44f5-9b11-4ba2fe60a310","Type":"ContainerDied","Data":"385889e55e0511d223458cf628f69c8402c10c2aff3f4cb6c512fd000fb06e31"}
Sep 16 14:37:53.018385 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.018090    2558 scope.go:117] "RemoveContainer" containerID="f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"
Sep 16 14:37:53.025079 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.025061    2558 scope.go:117] "RemoveContainer" containerID="0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"
Sep 16 14:37:53.031494 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.031478    2558 scope.go:117] "RemoveContainer" containerID="5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"
Sep 16 14:37:53.034369 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.034147    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:37:53.039039 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.039018    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:37:53.039748 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.039724    2558 scope.go:117] "RemoveContainer" containerID="11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"
Sep 16 14:37:53.046643 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.046621    2558 scope.go:117] "RemoveContainer" containerID="3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"
Sep 16 14:37:53.053623 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.053602    2558 scope.go:117] "RemoveContainer" containerID="dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"
Sep 16 14:37:53.060347 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.060330    2558 scope.go:117] "RemoveContainer" containerID="ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"
Sep 16 14:37:53.060632 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:37:53.060604    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": container with ID starting with ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca not found: ID does not exist" containerID="ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"
Sep 16 14:37:53.060721 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.060638    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"} err="failed to get container status \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": rpc error: code = NotFound desc = could not find container \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": container with ID starting with ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca not found: ID does not exist"
Sep 16 14:37:53.060721 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.060664    2558 scope.go:117] "RemoveContainer" containerID="f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"
Sep 16 14:37:53.060931 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:37:53.060908    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": container with ID starting with f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a not found: ID does not exist" containerID="f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"
Sep 16 14:37:53.060978 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.060940    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"} err="failed to get container status \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": rpc error: code = NotFound desc = could not find container \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": container with ID starting with f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a not found: ID does not exist"
Sep 16 14:37:53.060978 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.060962    2558 scope.go:117] "RemoveContainer" containerID="0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"
Sep 16 14:37:53.061296 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:37:53.061272    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": container with ID starting with 0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072 not found: ID does not exist" containerID="0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"
Sep 16 14:37:53.061399 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.061302    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"} err="failed to get container status \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": rpc error: code = NotFound desc = could not find container \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": container with ID starting with 0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072 not found: ID does not exist"
Sep 16 14:37:53.061399 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.061321    2558 scope.go:117] "RemoveContainer" containerID="5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"
Sep 16 14:37:53.061617 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:37:53.061600    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": container with ID starting with 5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e not found: ID does not exist" containerID="5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"
Sep 16 14:37:53.061687 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.061626    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"} err="failed to get container status \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": rpc error: code = NotFound desc = could not find container \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": container with ID starting with 5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e not found: ID does not exist"
Sep 16 14:37:53.061687 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.061647    2558 scope.go:117] "RemoveContainer" containerID="11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"
Sep 16 14:37:53.061905 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:37:53.061890    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": container with ID starting with 11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5 not found: ID does not exist" containerID="11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"
Sep 16 14:37:53.061943 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.061918    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"} err="failed to get container status \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": rpc error: code = NotFound desc = could not find container \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": container with ID starting with 11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5 not found: ID does not exist"
Sep 16 14:37:53.061943 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.061933    2558 scope.go:117] "RemoveContainer" containerID="3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"
Sep 16 14:37:53.062187 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:37:53.062157    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": container with ID starting with 3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983 not found: ID does not exist" containerID="3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"
Sep 16 14:37:53.062243 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.062191    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"} err="failed to get container status \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": rpc error: code = NotFound desc = could not find container \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": container with ID starting with 3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983 not found: ID does not exist"
Sep 16 14:37:53.062243 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.062206    2558 scope.go:117] "RemoveContainer" containerID="dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"
Sep 16 14:37:53.062424 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:37:53.062407    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": container with ID starting with dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217 not found: ID does not exist" containerID="dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"
Sep 16 14:37:53.062483 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.062433    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"} err="failed to get container status \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": rpc error: code = NotFound desc = could not find container \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": container with ID starting with dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217 not found: ID does not exist"
Sep 16 14:37:53.062483 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.062453    2558 scope.go:117] "RemoveContainer" containerID="ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"
Sep 16 14:37:53.062672 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.062656    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"} err="failed to get container status \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": rpc error: code = NotFound desc = could not find container \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": container with ID starting with ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca not found: ID does not exist"
Sep 16 14:37:53.062716 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.062673    2558 scope.go:117] "RemoveContainer" containerID="f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"
Sep 16 14:37:53.062867 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.062848    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"} err="failed to get container status \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": rpc error: code = NotFound desc = could not find container \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": container with ID starting with f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a not found: ID does not exist"
Sep 16 14:37:53.062932 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.062868    2558 scope.go:117] "RemoveContainer" containerID="0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"
Sep 16 14:37:53.063077 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.063062    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"} err="failed to get container status \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": rpc error: code = NotFound desc = could not find container \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": container with ID starting with 0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072 not found: ID does not exist"
Sep 16 14:37:53.063129 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.063077    2558 scope.go:117] "RemoveContainer" containerID="5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"
Sep 16 14:37:53.063279 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.063263    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"} err="failed to get container status \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": rpc error: code = NotFound desc = could not find container \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": container with ID starting with 5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e not found: ID does not exist"
Sep 16 14:37:53.063330 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.063279    2558 scope.go:117] "RemoveContainer" containerID="11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"
Sep 16 14:37:53.063471 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.063456    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"} err="failed to get container status \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": rpc error: code = NotFound desc = could not find container \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": container with ID starting with 11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5 not found: ID does not exist"
Sep 16 14:37:53.063510 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.063472    2558 scope.go:117] "RemoveContainer" containerID="3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"
Sep 16 14:37:53.063673 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.063656    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"} err="failed to get container status \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": rpc error: code = NotFound desc = could not find container \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": container with ID starting with 3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983 not found: ID does not exist"
Sep 16 14:37:53.063712 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.063674    2558 scope.go:117] "RemoveContainer" containerID="dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"
Sep 16 14:37:53.063889 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.063874    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"} err="failed to get container status \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": rpc error: code = NotFound desc = could not find container \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": container with ID starting with dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217 not found: ID does not exist"
Sep 16 14:37:53.063926 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.063889    2558 scope.go:117] "RemoveContainer" containerID="ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"
Sep 16 14:37:53.064085 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.064069    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"} err="failed to get container status \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": rpc error: code = NotFound desc = could not find container \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": container with ID starting with ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca not found: ID does not exist"
Sep 16 14:37:53.064123 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.064087    2558 scope.go:117] "RemoveContainer" containerID="f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"
Sep 16 14:37:53.064336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.064318    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"} err="failed to get container status \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": rpc error: code = NotFound desc = could not find container \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": container with ID starting with f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a not found: ID does not exist"
Sep 16 14:37:53.064336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.064335    2558 scope.go:117] "RemoveContainer" containerID="0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"
Sep 16 14:37:53.064543 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.064526    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"} err="failed to get container status \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": rpc error: code = NotFound desc = could not find container \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": container with ID starting with 0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072 not found: ID does not exist"
Sep 16 14:37:53.064584 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.064543    2558 scope.go:117] "RemoveContainer" containerID="5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"
Sep 16 14:37:53.064749 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.064734    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"} err="failed to get container status \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": rpc error: code = NotFound desc = could not find container \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": container with ID starting with 5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e not found: ID does not exist"
Sep 16 14:37:53.064749 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.064748    2558 scope.go:117] "RemoveContainer" containerID="11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"
Sep 16 14:37:53.064951 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.064934    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"} err="failed to get container status \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": rpc error: code = NotFound desc = could not find container \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": container with ID starting with 11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5 not found: ID does not exist"
Sep 16 14:37:53.064988 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.064951    2558 scope.go:117] "RemoveContainer" containerID="3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"
Sep 16 14:37:53.065147 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.065132    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"} err="failed to get container status \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": rpc error: code = NotFound desc = could not find container \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": container with ID starting with 3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983 not found: ID does not exist"
Sep 16 14:37:53.065147 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.065146    2558 scope.go:117] "RemoveContainer" containerID="dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"
Sep 16 14:37:53.065341 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.065325    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"} err="failed to get container status \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": rpc error: code = NotFound desc = could not find container \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": container with ID starting with dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217 not found: ID does not exist"
Sep 16 14:37:53.065386 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.065341    2558 scope.go:117] "RemoveContainer" containerID="ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"
Sep 16 14:37:53.065569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.065548    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"} err="failed to get container status \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": rpc error: code = NotFound desc = could not find container \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": container with ID starting with ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca not found: ID does not exist"
Sep 16 14:37:53.065569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.065567    2558 scope.go:117] "RemoveContainer" containerID="f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"
Sep 16 14:37:53.065802 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.065786    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"} err="failed to get container status \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": rpc error: code = NotFound desc = could not find container \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": container with ID starting with f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a not found: ID does not exist"
Sep 16 14:37:53.065849 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.065803    2558 scope.go:117] "RemoveContainer" containerID="0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"
Sep 16 14:37:53.066018 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.066002    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"} err="failed to get container status \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": rpc error: code = NotFound desc = could not find container \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": container with ID starting with 0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072 not found: ID does not exist"
Sep 16 14:37:53.066067 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.066019    2558 scope.go:117] "RemoveContainer" containerID="5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"
Sep 16 14:37:53.066259 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.066239    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"} err="failed to get container status \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": rpc error: code = NotFound desc = could not find container \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": container with ID starting with 5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e not found: ID does not exist"
Sep 16 14:37:53.066304 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.066260    2558 scope.go:117] "RemoveContainer" containerID="11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"
Sep 16 14:37:53.066472 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.066456    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"} err="failed to get container status \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": rpc error: code = NotFound desc = could not find container \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": container with ID starting with 11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5 not found: ID does not exist"
Sep 16 14:37:53.066515 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.066473    2558 scope.go:117] "RemoveContainer" containerID="3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"
Sep 16 14:37:53.066715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.066698    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"} err="failed to get container status \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": rpc error: code = NotFound desc = could not find container \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": container with ID starting with 3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983 not found: ID does not exist"
Sep 16 14:37:53.066761 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.066717    2558 scope.go:117] "RemoveContainer" containerID="dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"
Sep 16 14:37:53.066922 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.066901    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"} err="failed to get container status \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": rpc error: code = NotFound desc = could not find container \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": container with ID starting with dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217 not found: ID does not exist"
Sep 16 14:37:53.066967 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.066924    2558 scope.go:117] "RemoveContainer" containerID="ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"
Sep 16 14:37:53.067123 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.067105    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"} err="failed to get container status \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": rpc error: code = NotFound desc = could not find container \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": container with ID starting with ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca not found: ID does not exist"
Sep 16 14:37:53.067175 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.067125    2558 scope.go:117] "RemoveContainer" containerID="f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"
Sep 16 14:37:53.067349 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.067333    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"} err="failed to get container status \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": rpc error: code = NotFound desc = could not find container \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": container with ID starting with f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a not found: ID does not exist"
Sep 16 14:37:53.067393 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.067349    2558 scope.go:117] "RemoveContainer" containerID="0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"
Sep 16 14:37:53.067566 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.067549    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"} err="failed to get container status \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": rpc error: code = NotFound desc = could not find container \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": container with ID starting with 0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072 not found: ID does not exist"
Sep 16 14:37:53.067614 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.067567    2558 scope.go:117] "RemoveContainer" containerID="5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"
Sep 16 14:37:53.067785 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.067766    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"} err="failed to get container status \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": rpc error: code = NotFound desc = could not find container \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": container with ID starting with 5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e not found: ID does not exist"
Sep 16 14:37:53.067827 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.067786    2558 scope.go:117] "RemoveContainer" containerID="11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"
Sep 16 14:37:53.067983 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.067961    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"} err="failed to get container status \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": rpc error: code = NotFound desc = could not find container \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": container with ID starting with 11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5 not found: ID does not exist"
Sep 16 14:37:53.068023 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.067985    2558 scope.go:117] "RemoveContainer" containerID="3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"
Sep 16 14:37:53.068243 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.068219    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"} err="failed to get container status \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": rpc error: code = NotFound desc = could not find container \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": container with ID starting with 3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983 not found: ID does not exist"
Sep 16 14:37:53.068406 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.068244    2558 scope.go:117] "RemoveContainer" containerID="dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"
Sep 16 14:37:53.068655 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.068599    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"} err="failed to get container status \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": rpc error: code = NotFound desc = could not find container \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": container with ID starting with dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217 not found: ID does not exist"
Sep 16 14:37:53.068655 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.068627    2558 scope.go:117] "RemoveContainer" containerID="ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"
Sep 16 14:37:53.068949 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.068927    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca"} err="failed to get container status \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": rpc error: code = NotFound desc = could not find container \"ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca\": container with ID starting with ec616ae0ffd3f76eb43cfc65abec5125d898919de0c67f9c325f735fd3748aca not found: ID does not exist"
Sep 16 14:37:53.069051 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.068951    2558 scope.go:117] "RemoveContainer" containerID="f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"
Sep 16 14:37:53.069260 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.069230    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a"} err="failed to get container status \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": rpc error: code = NotFound desc = could not find container \"f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a\": container with ID starting with f36edc0a43da48d10a587a7c1d220981f03ea877352b4c3af19eecb38b54d09a not found: ID does not exist"
Sep 16 14:37:53.069260 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.069253    2558 scope.go:117] "RemoveContainer" containerID="0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"
Sep 16 14:37:53.069546 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.069525    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072"} err="failed to get container status \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": rpc error: code = NotFound desc = could not find container \"0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072\": container with ID starting with 0e05ddebc650674741f3bfca5c9dfeec3cee76b6ce68d128d64211048c503072 not found: ID does not exist"
Sep 16 14:37:53.069608 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.069546    2558 scope.go:117] "RemoveContainer" containerID="5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"
Sep 16 14:37:53.069805 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.069782    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e"} err="failed to get container status \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": rpc error: code = NotFound desc = could not find container \"5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e\": container with ID starting with 5d70c600831c5d0dba71f121fd73cdd6b43786fd9d832a8c641bdfea6db40f0e not found: ID does not exist"
Sep 16 14:37:53.069885 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.069807    2558 scope.go:117] "RemoveContainer" containerID="11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"
Sep 16 14:37:53.070023 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070007    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5"} err="failed to get container status \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": rpc error: code = NotFound desc = could not find container \"11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5\": container with ID starting with 11e4c8f0dd1c8ba60c55014790fcc2d2de0f1c74f781571ad7d8d8dd07cf76c5 not found: ID does not exist"
Sep 16 14:37:53.070082 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070025    2558 scope.go:117] "RemoveContainer" containerID="3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"
Sep 16 14:37:53.070282 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070266    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983"} err="failed to get container status \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": rpc error: code = NotFound desc = could not find container \"3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983\": container with ID starting with 3dbc9888f2cb6c208fc895893f1b3a56339bbc930a11eca0efc9e0244517a983 not found: ID does not exist"
Sep 16 14:37:53.070334 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070283    2558 scope.go:117] "RemoveContainer" containerID="dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"
Sep 16 14:37:53.070334 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070283    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:37:53.070571 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070549    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217"} err="failed to get container status \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": rpc error: code = NotFound desc = could not find container \"dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217\": container with ID starting with dab430891cc6e90b36ecea1fbedecf8765cd4a1c60575c4ba16195863286f217 not found: ID does not exist"
Sep 16 14:37:53.070612 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070587    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="init-config-reloader"
Sep 16 14:37:53.070612 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070599    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="init-config-reloader"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070613    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="prometheus"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070619    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="prometheus"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070626    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy-web"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070632    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy-web"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070638    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="config-reloader"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070643    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="config-reloader"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070648    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="thanos-sidecar"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070654    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="thanos-sidecar"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070660    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy-thanos"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070664    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy-thanos"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070674    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy"
Sep 16 14:37:53.070681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070680    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy"
Sep 16 14:37:53.071048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070720    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy-thanos"
Sep 16 14:37:53.071048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070727    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="prometheus"
Sep 16 14:37:53.071048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070734    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy"
Sep 16 14:37:53.071048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070739    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="config-reloader"
Sep 16 14:37:53.071048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070744    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="thanos-sidecar"
Sep 16 14:37:53.071048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.070749    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" containerName="kube-rbac-proxy-web"
Sep 16 14:37:53.075499 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.075481    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.079261 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.079231    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 14:37:53.079361 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.079294    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 14:37:53.079691 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.079671    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 14:37:53.079814 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.079673    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 14:37:53.079814 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.079736    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 14:37:53.079814 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.079680    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 14:37:53.079814 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.079810    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 14:37:53.079814 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.079828    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 14:37:53.080057 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.079946    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 14:37:53.080057 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.080016    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 14:37:53.080057 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.080030    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 14:37:53.080188 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.080077    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-v6wvg\""
Sep 16 14:37:53.080188 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.080031    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2i78j051k51m2\""
Sep 16 14:37:53.081292 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.081272    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 14:37:53.086552 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.086535    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 14:37:53.089101 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.089085    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 14:37:53.091482 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.091462    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:37:53.240557 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240468    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9b3d956-68a6-4e23-b10d-4aef5c396f70-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.240557 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240497    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.240557 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240525    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.240830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240630    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.240830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240667    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.240830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240689    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.240830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240721    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-config\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.240830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240740    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxvv\" (UniqueName: \"kubernetes.io/projected/a9b3d956-68a6-4e23-b10d-4aef5c396f70-kube-api-access-zvxvv\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.240830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240760    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.240830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240812    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.241195 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240848    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.241195 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240879    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.241195 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9b3d956-68a6-4e23-b10d-4aef5c396f70-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.241195 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240936    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.241195 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240970    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.241195 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.240996    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.241195 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.241022    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.241195 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.241046    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.241195 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.241072    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.241195 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.241101    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.297221 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.297190    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4107a59-2d99-44f5-9b11-4ba2fe60a310" path="/var/lib/kubelet/pods/a4107a59-2d99-44f5-9b11-4ba2fe60a310/volumes"
Sep 16 14:37:53.341739 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.341694    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.341834 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.341765    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.341834 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.341799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342004 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.341970    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-config\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342063 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxvv\" (UniqueName: \"kubernetes.io/projected/a9b3d956-68a6-4e23-b10d-4aef5c396f70-kube-api-access-zvxvv\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342118 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342067    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342118 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342090    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342251 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342117    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342251 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342146    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342251 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342194    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9b3d956-68a6-4e23-b10d-4aef5c396f70-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342251 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342221    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342251 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342247    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342619 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342619 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342288    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342619 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342298    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.342619 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.342324    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.343351 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.343325    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.343848 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.343824    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.345206 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.344914    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.345206 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.344933    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-config\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.345206 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.344964    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.345206 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.345000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.345206 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.345037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9b3d956-68a6-4e23-b10d-4aef5c396f70-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.345206 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.345064    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.345206 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.345090    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.345206 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.345093    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.345206 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.345107    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.345945 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.345247    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9b3d956-68a6-4e23-b10d-4aef5c396f70-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.345945 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.345733    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.346008 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.345985    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.346315 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.346286    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.346729 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.346701    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.346953 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.346931    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.347222 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.347202    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.347666 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.347636    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.347666 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.347650    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9b3d956-68a6-4e23-b10d-4aef5c396f70-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.347863 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.347840    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.348259 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.348238    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9b3d956-68a6-4e23-b10d-4aef5c396f70-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.348313 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.348261    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9b3d956-68a6-4e23-b10d-4aef5c396f70-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.349983 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.349966    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxvv\" (UniqueName: \"kubernetes.io/projected/a9b3d956-68a6-4e23-b10d-4aef5c396f70-kube-api-access-zvxvv\") pod \"prometheus-k8s-0\" (UID: \"a9b3d956-68a6-4e23-b10d-4aef5c396f70\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.385062 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.385040    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:37:53.666818 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:53.666791    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 14:37:53.667730 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:37:53.667708    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b3d956_68a6_4e23_b10d_4aef5c396f70.slice/crio-5114ba525f5bf40a5bb1bf8a0eceb4a64afe086afd177aeca0933613a8eb4ad6 WatchSource:0}: Error finding container 5114ba525f5bf40a5bb1bf8a0eceb4a64afe086afd177aeca0933613a8eb4ad6: Status 404 returned error can't find the container with id 5114ba525f5bf40a5bb1bf8a0eceb4a64afe086afd177aeca0933613a8eb4ad6
Sep 16 14:37:54.016633 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:54.016545    2558 generic.go:356] "Generic (PLEG): container finished" podID="a9b3d956-68a6-4e23-b10d-4aef5c396f70" containerID="85faeca9253c948a05ee76b8c595e292956147b61bee7eeb47a8160ff613eb81" exitCode=0
Sep 16 14:37:54.016633 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:54.016587    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9b3d956-68a6-4e23-b10d-4aef5c396f70","Type":"ContainerDied","Data":"85faeca9253c948a05ee76b8c595e292956147b61bee7eeb47a8160ff613eb81"}
Sep 16 14:37:54.017029 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:54.016651    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9b3d956-68a6-4e23-b10d-4aef5c396f70","Type":"ContainerStarted","Data":"5114ba525f5bf40a5bb1bf8a0eceb4a64afe086afd177aeca0933613a8eb4ad6"}
Sep 16 14:37:55.023910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:55.023874    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9b3d956-68a6-4e23-b10d-4aef5c396f70","Type":"ContainerStarted","Data":"20b3599ad652009f094d1f35ab7201d3eb84eefd63a7ac23c31ad4ae6bf1a5af"}
Sep 16 14:37:55.024306 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:55.023954    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9b3d956-68a6-4e23-b10d-4aef5c396f70","Type":"ContainerStarted","Data":"6e53290c95c5aff46c1bb3f6c78a48103c5746544d06beec2f08e68a4eccdbd2"}
Sep 16 14:37:55.024306 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:55.023991    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9b3d956-68a6-4e23-b10d-4aef5c396f70","Type":"ContainerStarted","Data":"2b7f11de85db53018b03ff1f7680fe1836a3fe2e711b367dbf473f5b700df36c"}
Sep 16 14:37:55.024306 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:55.024030    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9b3d956-68a6-4e23-b10d-4aef5c396f70","Type":"ContainerStarted","Data":"910ab891eb13093ac25e24440f08610f1f927a03b72f288fd6da1fcee71b8230"}
Sep 16 14:37:55.024306 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:55.024069    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9b3d956-68a6-4e23-b10d-4aef5c396f70","Type":"ContainerStarted","Data":"1e66a431bd800f31ac62fcfc7d6a7c27970127a18e11769702d4ce4514b21a58"}
Sep 16 14:37:55.024306 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:55.024106    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9b3d956-68a6-4e23-b10d-4aef5c396f70","Type":"ContainerStarted","Data":"897b593444e197f459345d87d875786753fba6fb333e605713a8bef6f2de4c08"}
Sep 16 14:37:55.056285 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:55.056232    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.038571163 podStartE2EDuration="2.056215146s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.643983ms" isStatefulPod=false podCreationTimestamp="2026-09-16 14:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:37:55.054736216 +0000 UTC m=+152.208814503" watchObservedRunningTime="2026-09-16 14:37:55.056215146 +0000 UTC m=+152.210293433"
Sep 16 14:37:58.385503 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:37:58.385458    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:38:53.386224 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:38:53.386195    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:38:53.401051 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:38:53.401027    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:38:54.243039 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:38:54.243013    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 14:40:23.157263 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:23.157235    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:40:23.161580 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:23.161560    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:40:23.163859 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:23.163839    2558 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 14:40:34.620427 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.620393    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4"]
Sep 16 14:40:34.623494 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.623478    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4"
Sep 16 14:40:34.627273 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.627251    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5zz7j\""
Sep 16 14:40:34.628436 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.628416    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 14:40:34.628588 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.628429    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 14:40:34.643741 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.643718    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4"]
Sep 16 14:40:34.704148 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.704126    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f56d\" (UniqueName: \"kubernetes.io/projected/c21dffce-72ca-4a0a-bd4d-46e27132af1f-kube-api-access-7f56d\") pod \"cert-manager-cainjector-559cdd8c87-jrtc4\" (UID: \"c21dffce-72ca-4a0a-bd4d-46e27132af1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4"
Sep 16 14:40:34.704262 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.704158    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c21dffce-72ca-4a0a-bd4d-46e27132af1f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jrtc4\" (UID: \"c21dffce-72ca-4a0a-bd4d-46e27132af1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4"
Sep 16 14:40:34.805398 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.805373    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c21dffce-72ca-4a0a-bd4d-46e27132af1f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jrtc4\" (UID: \"c21dffce-72ca-4a0a-bd4d-46e27132af1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4"
Sep 16 14:40:34.805491 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.805436    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f56d\" (UniqueName: \"kubernetes.io/projected/c21dffce-72ca-4a0a-bd4d-46e27132af1f-kube-api-access-7f56d\") pod \"cert-manager-cainjector-559cdd8c87-jrtc4\" (UID: \"c21dffce-72ca-4a0a-bd4d-46e27132af1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4"
Sep 16 14:40:34.814206 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.814152    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c21dffce-72ca-4a0a-bd4d-46e27132af1f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jrtc4\" (UID: \"c21dffce-72ca-4a0a-bd4d-46e27132af1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4"
Sep 16 14:40:34.814410 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.814395    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f56d\" (UniqueName: \"kubernetes.io/projected/c21dffce-72ca-4a0a-bd4d-46e27132af1f-kube-api-access-7f56d\") pod \"cert-manager-cainjector-559cdd8c87-jrtc4\" (UID: \"c21dffce-72ca-4a0a-bd4d-46e27132af1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4"
Sep 16 14:40:34.941481 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:34.941417    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4"
Sep 16 14:40:35.214964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:35.214818    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4"]
Sep 16 14:40:35.217467 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:40:35.217437    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc21dffce_72ca_4a0a_bd4d_46e27132af1f.slice/crio-ed2fbea10a99f49e336e5609c2f69079c16bc810402918a1004a784e2dcc5979 WatchSource:0}: Error finding container ed2fbea10a99f49e336e5609c2f69079c16bc810402918a1004a784e2dcc5979: Status 404 returned error can't find the container with id ed2fbea10a99f49e336e5609c2f69079c16bc810402918a1004a784e2dcc5979
Sep 16 14:40:35.219458 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:35.219445    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:40:35.587137 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:35.587105    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4" event={"ID":"c21dffce-72ca-4a0a-bd4d-46e27132af1f","Type":"ContainerStarted","Data":"ed2fbea10a99f49e336e5609c2f69079c16bc810402918a1004a784e2dcc5979"}
Sep 16 14:40:38.599818 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:38.599790    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4" event={"ID":"c21dffce-72ca-4a0a-bd4d-46e27132af1f","Type":"ContainerStarted","Data":"3b824fbee9805ed6221a67c05eca71d7ddc843dea22d2003b8d14db256b684ab"}
Sep 16 14:40:38.616808 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:38.616760    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jrtc4" podStartSLOduration=1.31211163 podStartE2EDuration="4.616743549s" totalImagesPullingTime="3.304631919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:40:38.616190702 +0000 UTC m=+315.770269006" watchObservedRunningTime="2026-09-16 14:40:38.616743549 +0000 UTC m=+315.770821837"
Sep 16 14:40:41.825128 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:41.825088    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6"]
Sep 16 14:40:41.828153 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:41.828131    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6"
Sep 16 14:40:41.831758 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:41.831736    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 14:40:41.831859 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:41.831783    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 14:40:41.832972 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:41.832950    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-f26jb\""
Sep 16 14:40:41.846441 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:41.846419    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6"]
Sep 16 14:40:41.970348 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:41.970312    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbh9s\" (UniqueName: \"kubernetes.io/projected/c6d26148-68b6-4557-94fa-1c4f60ac8bdb-kube-api-access-fbh9s\") pod \"openshift-lws-operator-fd8ccff4c-mq6t6\" (UID: \"c6d26148-68b6-4557-94fa-1c4f60ac8bdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6"
Sep 16 14:40:41.970509 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:41.970358    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6d26148-68b6-4557-94fa-1c4f60ac8bdb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mq6t6\" (UID: \"c6d26148-68b6-4557-94fa-1c4f60ac8bdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6"
Sep 16 14:40:42.071101 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:42.071060    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbh9s\" (UniqueName: \"kubernetes.io/projected/c6d26148-68b6-4557-94fa-1c4f60ac8bdb-kube-api-access-fbh9s\") pod \"openshift-lws-operator-fd8ccff4c-mq6t6\" (UID: \"c6d26148-68b6-4557-94fa-1c4f60ac8bdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6"
Sep 16 14:40:42.071296 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:42.071110    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6d26148-68b6-4557-94fa-1c4f60ac8bdb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mq6t6\" (UID: \"c6d26148-68b6-4557-94fa-1c4f60ac8bdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6"
Sep 16 14:40:42.071462 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:42.071446    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6d26148-68b6-4557-94fa-1c4f60ac8bdb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-mq6t6\" (UID: \"c6d26148-68b6-4557-94fa-1c4f60ac8bdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6"
Sep 16 14:40:42.080596 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:42.080529    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbh9s\" (UniqueName: \"kubernetes.io/projected/c6d26148-68b6-4557-94fa-1c4f60ac8bdb-kube-api-access-fbh9s\") pod \"openshift-lws-operator-fd8ccff4c-mq6t6\" (UID: \"c6d26148-68b6-4557-94fa-1c4f60ac8bdb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6"
Sep 16 14:40:42.136970 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:42.136931    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6"
Sep 16 14:40:42.410125 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:42.410047    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6"]
Sep 16 14:40:42.413900 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:40:42.413875    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6d26148_68b6_4557_94fa_1c4f60ac8bdb.slice/crio-0cc6b6c40c878931ae05726012f7e88dad0b69cb855dee68b020e3991f87c7e8 WatchSource:0}: Error finding container 0cc6b6c40c878931ae05726012f7e88dad0b69cb855dee68b020e3991f87c7e8: Status 404 returned error can't find the container with id 0cc6b6c40c878931ae05726012f7e88dad0b69cb855dee68b020e3991f87c7e8
Sep 16 14:40:42.615895 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:42.615859    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6" event={"ID":"c6d26148-68b6-4557-94fa-1c4f60ac8bdb","Type":"ContainerStarted","Data":"0cc6b6c40c878931ae05726012f7e88dad0b69cb855dee68b020e3991f87c7e8"}
Sep 16 14:40:44.627869 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:44.627835    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6" event={"ID":"c6d26148-68b6-4557-94fa-1c4f60ac8bdb","Type":"ContainerStarted","Data":"39570d3caeb10ae28eae90b6feb27c5f9a565527a93b5d01cbf259dee91653e1"}
Sep 16 14:40:44.646802 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:40:44.646736    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-mq6t6" podStartSLOduration=1.534641126 podStartE2EDuration="3.646715962s" totalImagesPullingTime="2.112074836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:40:44.644561181 +0000 UTC m=+321.798639469" watchObservedRunningTime="2026-09-16 14:40:44.646715962 +0000 UTC m=+321.800794253"
Sep 16 14:41:02.136554 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.136525    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"]
Sep 16 14:41:02.139715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.139699    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.142735 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.142709    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcf5v\" (UniqueName: \"kubernetes.io/projected/f2b65df2-60ea-42f6-b57e-913ada173141-kube-api-access-lcf5v\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.142920 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.142770    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f2b65df2-60ea-42f6-b57e-913ada173141-manager-config\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.142920 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.142855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2b65df2-60ea-42f6-b57e-913ada173141-cert\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.142920 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.142892    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2b65df2-60ea-42f6-b57e-913ada173141-metrics-cert\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.143964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.143945    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 14:41:02.143964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.143956    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ql22x\""
Sep 16 14:41:02.144103 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.144003    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 14:41:02.144103 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.144066    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 14:41:02.153085 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.153063    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"]
Sep 16 14:41:02.243301 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.243215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2b65df2-60ea-42f6-b57e-913ada173141-cert\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.243301 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.243251    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2b65df2-60ea-42f6-b57e-913ada173141-metrics-cert\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.243301 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.243296    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcf5v\" (UniqueName: \"kubernetes.io/projected/f2b65df2-60ea-42f6-b57e-913ada173141-kube-api-access-lcf5v\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.243581 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.243325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f2b65df2-60ea-42f6-b57e-913ada173141-manager-config\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.243975 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.243954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f2b65df2-60ea-42f6-b57e-913ada173141-manager-config\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.245684 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.245665    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2b65df2-60ea-42f6-b57e-913ada173141-cert\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.245783 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.245767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f2b65df2-60ea-42f6-b57e-913ada173141-metrics-cert\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.258296 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.258267    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcf5v\" (UniqueName: \"kubernetes.io/projected/f2b65df2-60ea-42f6-b57e-913ada173141-kube-api-access-lcf5v\") pod \"lws-controller-manager-8787d76cb-lt6k8\" (UID: \"f2b65df2-60ea-42f6-b57e-913ada173141\") " pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.449075 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.449036    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:02.731270 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.731238    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"]
Sep 16 14:41:02.734100 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:41:02.734075    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b65df2_60ea_42f6_b57e_913ada173141.slice/crio-f04b3d71027fae1cef7bcde809253c707276a7482a11556bc8edccd5609d9ee6 WatchSource:0}: Error finding container f04b3d71027fae1cef7bcde809253c707276a7482a11556bc8edccd5609d9ee6: Status 404 returned error can't find the container with id f04b3d71027fae1cef7bcde809253c707276a7482a11556bc8edccd5609d9ee6
Sep 16 14:41:02.839700 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.839661    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"]
Sep 16 14:41:02.844077 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.844061    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.848375 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.848349    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/817c7241-feea-40c9-a095-af6e0ca69bb7-manifests\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.848508 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.848412    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mlpw\" (UniqueName: \"kubernetes.io/projected/817c7241-feea-40c9-a095-af6e0ca69bb7-kube-api-access-8mlpw\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.848508 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.848444    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 14:41:02.848508 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.848482    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/817c7241-feea-40c9-a095-af6e0ca69bb7-webhook-cert\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.848508 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.848505    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/817c7241-feea-40c9-a095-af6e0ca69bb7-tmp\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.848758 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.848532    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/817c7241-feea-40c9-a095-af6e0ca69bb7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.848758 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.848587    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2nhn5\""
Sep 16 14:41:02.849849 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.849829    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 14:41:02.849849 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.849843    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 14:41:02.850010 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.849881    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 14:41:02.948873 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.948837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mlpw\" (UniqueName: \"kubernetes.io/projected/817c7241-feea-40c9-a095-af6e0ca69bb7-kube-api-access-8mlpw\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.949053 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.948888    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/817c7241-feea-40c9-a095-af6e0ca69bb7-webhook-cert\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.949053 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.948909    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/817c7241-feea-40c9-a095-af6e0ca69bb7-tmp\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.949053 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.948933    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/817c7241-feea-40c9-a095-af6e0ca69bb7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.949053 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.948954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/817c7241-feea-40c9-a095-af6e0ca69bb7-manifests\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.949364 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.949344    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/817c7241-feea-40c9-a095-af6e0ca69bb7-manifests\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.951366 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.951343    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/817c7241-feea-40c9-a095-af6e0ca69bb7-tmp\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.951476 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.951449    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/817c7241-feea-40c9-a095-af6e0ca69bb7-webhook-cert\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.951526 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.951490    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/817c7241-feea-40c9-a095-af6e0ca69bb7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:02.962123 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:02.962107    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mlpw\" (UniqueName: \"kubernetes.io/projected/817c7241-feea-40c9-a095-af6e0ca69bb7-kube-api-access-8mlpw\") pod \"opendatahub-operator-controller-manager-846667779d-6p5jp\" (UID: \"817c7241-feea-40c9-a095-af6e0ca69bb7\") " pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:03.040054 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:03.040032    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"]
Sep 16 14:41:03.154200 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:03.154105    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:03.437029 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:03.436942    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"]
Sep 16 14:41:03.441990 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:41:03.441963    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817c7241_feea_40c9_a095_af6e0ca69bb7.slice/crio-389065f92918df71a2c1449fbc0936e87e81c1b99b61aa7b39747cc6a074afd0 WatchSource:0}: Error finding container 389065f92918df71a2c1449fbc0936e87e81c1b99b61aa7b39747cc6a074afd0: Status 404 returned error can't find the container with id 389065f92918df71a2c1449fbc0936e87e81c1b99b61aa7b39747cc6a074afd0
Sep 16 14:41:03.699929 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:03.699839    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp" event={"ID":"817c7241-feea-40c9-a095-af6e0ca69bb7","Type":"ContainerStarted","Data":"389065f92918df71a2c1449fbc0936e87e81c1b99b61aa7b39747cc6a074afd0"}
Sep 16 14:41:03.701315 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:03.701288    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8" event={"ID":"f2b65df2-60ea-42f6-b57e-913ada173141","Type":"ContainerStarted","Data":"f04b3d71027fae1cef7bcde809253c707276a7482a11556bc8edccd5609d9ee6"}
Sep 16 14:41:04.708572 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:04.708449    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8" event={"ID":"f2b65df2-60ea-42f6-b57e-913ada173141","Type":"ContainerStarted","Data":"020598f8383d4f23935ea33bbb9c07772d282dd7931d5f7a72e21f4371fcc566"}
Sep 16 14:41:04.709019 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:04.708602    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:04.731862 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:04.731803    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8" podStartSLOduration=1.133526239 podStartE2EDuration="2.731781932s" totalImagesPullingTime="1.598255693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:41:04.730635969 +0000 UTC m=+341.884714257" watchObservedRunningTime="2026-09-16 14:41:04.731781932 +0000 UTC m=+341.885860221"
Sep 16 14:41:07.725775 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:07.725737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp" event={"ID":"817c7241-feea-40c9-a095-af6e0ca69bb7","Type":"ContainerStarted","Data":"01aed3d3d57cd61c172fb2899d6bae4724c4b44ebac6ca4fcd9724f520f4918f"}
Sep 16 14:41:08.731178 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:08.731133    2558 generic.go:356] "Generic (PLEG): container finished" podID="817c7241-feea-40c9-a095-af6e0ca69bb7" containerID="01aed3d3d57cd61c172fb2899d6bae4724c4b44ebac6ca4fcd9724f520f4918f" exitCode=0
Sep 16 14:41:08.731542 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:08.731221    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp" event={"ID":"817c7241-feea-40c9-a095-af6e0ca69bb7","Type":"ContainerDied","Data":"01aed3d3d57cd61c172fb2899d6bae4724c4b44ebac6ca4fcd9724f520f4918f"}
Sep 16 14:41:09.737188 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:09.737129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp" event={"ID":"817c7241-feea-40c9-a095-af6e0ca69bb7","Type":"ContainerStarted","Data":"8b45434500d022b1545f374a4ad7757ed6fb3cd6a616cf6fc81e451bab4b38e7"}
Sep 16 14:41:09.737653 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:09.737242    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:09.775640 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:09.775588    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp" podStartSLOduration=2.995614663 podStartE2EDuration="7.775571731s" totalImagesPullingTime="3.742859931s" totalInitContainerRuntime="1.037097137s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:41:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:41:09.774138915 +0000 UTC m=+346.928217202" watchObservedRunningTime="2026-09-16 14:41:09.775571731 +0000 UTC m=+346.929650080"
Sep 16 14:41:15.718782 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:15.718750    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8787d76cb-lt6k8"
Sep 16 14:41:20.743946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:20.743918    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-846667779d-6p5jp"
Sep 16 14:41:23.878935 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:23.878900    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"]
Sep 16 14:41:23.884251 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:23.884227    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"
Sep 16 14:41:23.887108 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:23.887085    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 14:41:23.887263 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:23.887222    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cz257\""
Sep 16 14:41:23.887263 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:23.887243    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 14:41:23.888238 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:23.888222    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 14:41:23.888319 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:23.888286    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 14:41:23.895179 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:23.895143    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"]
Sep 16 14:41:23.936333 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:23.936306    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69122b7c-a0c3-4304-b291-cd6dbda74797-tls-certs\") pod \"kube-auth-proxy-7d79898bc9-j4nvg\" (UID: \"69122b7c-a0c3-4304-b291-cd6dbda74797\") " pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"
Sep 16 14:41:23.936506 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:23.936354    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69122b7c-a0c3-4304-b291-cd6dbda74797-tmp\") pod \"kube-auth-proxy-7d79898bc9-j4nvg\" (UID: \"69122b7c-a0c3-4304-b291-cd6dbda74797\") " pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"
Sep 16 14:41:23.936506 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:23.936445    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxgh7\" (UniqueName: \"kubernetes.io/projected/69122b7c-a0c3-4304-b291-cd6dbda74797-kube-api-access-hxgh7\") pod \"kube-auth-proxy-7d79898bc9-j4nvg\" (UID: \"69122b7c-a0c3-4304-b291-cd6dbda74797\") " pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"
Sep 16 14:41:24.037505 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:24.037465    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69122b7c-a0c3-4304-b291-cd6dbda74797-tmp\") pod \"kube-auth-proxy-7d79898bc9-j4nvg\" (UID: \"69122b7c-a0c3-4304-b291-cd6dbda74797\") " pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"
Sep 16 14:41:24.037672 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:24.037541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxgh7\" (UniqueName: \"kubernetes.io/projected/69122b7c-a0c3-4304-b291-cd6dbda74797-kube-api-access-hxgh7\") pod \"kube-auth-proxy-7d79898bc9-j4nvg\" (UID: \"69122b7c-a0c3-4304-b291-cd6dbda74797\") " pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"
Sep 16 14:41:24.037672 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:24.037634    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69122b7c-a0c3-4304-b291-cd6dbda74797-tls-certs\") pod \"kube-auth-proxy-7d79898bc9-j4nvg\" (UID: \"69122b7c-a0c3-4304-b291-cd6dbda74797\") " pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"
Sep 16 14:41:24.039832 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:24.039800    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69122b7c-a0c3-4304-b291-cd6dbda74797-tmp\") pod \"kube-auth-proxy-7d79898bc9-j4nvg\" (UID: \"69122b7c-a0c3-4304-b291-cd6dbda74797\") " pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"
Sep 16 14:41:24.039980 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:24.039964    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69122b7c-a0c3-4304-b291-cd6dbda74797-tls-certs\") pod \"kube-auth-proxy-7d79898bc9-j4nvg\" (UID: \"69122b7c-a0c3-4304-b291-cd6dbda74797\") " pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"
Sep 16 14:41:24.050332 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:24.050310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxgh7\" (UniqueName: \"kubernetes.io/projected/69122b7c-a0c3-4304-b291-cd6dbda74797-kube-api-access-hxgh7\") pod \"kube-auth-proxy-7d79898bc9-j4nvg\" (UID: \"69122b7c-a0c3-4304-b291-cd6dbda74797\") " pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"
Sep 16 14:41:24.193953 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:24.193867    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"
Sep 16 14:41:24.473800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:24.473774    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg"]
Sep 16 14:41:24.476148 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:41:24.476124    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69122b7c_a0c3_4304_b291_cd6dbda74797.slice/crio-859976a1b6a6893c1b623a929d79b26e8ba9320b0db96ca5add1df39016a5e35 WatchSource:0}: Error finding container 859976a1b6a6893c1b623a929d79b26e8ba9320b0db96ca5add1df39016a5e35: Status 404 returned error can't find the container with id 859976a1b6a6893c1b623a929d79b26e8ba9320b0db96ca5add1df39016a5e35
Sep 16 14:41:24.802859 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:24.802827    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg" event={"ID":"69122b7c-a0c3-4304-b291-cd6dbda74797","Type":"ContainerStarted","Data":"859976a1b6a6893c1b623a929d79b26e8ba9320b0db96ca5add1df39016a5e35"}
Sep 16 14:41:27.819428 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:27.819382    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg" event={"ID":"69122b7c-a0c3-4304-b291-cd6dbda74797","Type":"ContainerStarted","Data":"88fde67ff4889366e2151c563e30a6a6bf71f80da6cee6034f370249cd549649"}
Sep 16 14:41:27.837495 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:41:27.837446    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d79898bc9-j4nvg" podStartSLOduration=2.248976849 podStartE2EDuration="4.837428723s" totalImagesPullingTime="2.588451874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:41:27.836572617 +0000 UTC m=+364.990650905" watchObservedRunningTime="2026-09-16 14:41:27.837428723 +0000 UTC m=+364.991507010"
Sep 16 14:43:37.806669 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.806631    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"]
Sep 16 14:43:37.809832 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.809815    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"
Sep 16 14:43:37.813073 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.813046    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 14:43:37.813073 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.813062    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 14:43:37.813073 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.813069    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xtdb2\""
Sep 16 14:43:37.813297 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.813066    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 14:43:37.813883 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.813870    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 14:43:37.820012 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.819982    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"]
Sep 16 14:43:37.877704 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.877663    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8brgk\" (UniqueName: \"kubernetes.io/projected/7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99-kube-api-access-8brgk\") pod \"kuadrant-console-plugin-fbdd4f445-qxh5s\" (UID: \"7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"
Sep 16 14:43:37.877882 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.877719    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-qxh5s\" (UID: \"7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"
Sep 16 14:43:37.877882 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.877822    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-qxh5s\" (UID: \"7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"
Sep 16 14:43:37.978527 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.978499    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-qxh5s\" (UID: \"7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"
Sep 16 14:43:37.978706 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.978557    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-qxh5s\" (UID: \"7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"
Sep 16 14:43:37.978706 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.978586    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8brgk\" (UniqueName: \"kubernetes.io/projected/7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99-kube-api-access-8brgk\") pod \"kuadrant-console-plugin-fbdd4f445-qxh5s\" (UID: \"7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"
Sep 16 14:43:37.979210 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.979186    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-qxh5s\" (UID: \"7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"
Sep 16 14:43:37.981048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.981023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-qxh5s\" (UID: \"7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"
Sep 16 14:43:37.988801 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:37.988781    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8brgk\" (UniqueName: \"kubernetes.io/projected/7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99-kube-api-access-8brgk\") pod \"kuadrant-console-plugin-fbdd4f445-qxh5s\" (UID: \"7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"
Sep 16 14:43:38.121333 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:38.121249    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"
Sep 16 14:43:38.390246 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:38.390160    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s"]
Sep 16 14:43:38.393178 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:43:38.393137    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fed9b26_abf0_4d2a_8cbc_62fc3cbcad99.slice/crio-15664ab368765012c1c0a3e0c01f7359ee27643bb1d53062454cfe9477666790 WatchSource:0}: Error finding container 15664ab368765012c1c0a3e0c01f7359ee27643bb1d53062454cfe9477666790: Status 404 returned error can't find the container with id 15664ab368765012c1c0a3e0c01f7359ee27643bb1d53062454cfe9477666790
Sep 16 14:43:39.382496 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:39.382461    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s" event={"ID":"7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99","Type":"ContainerStarted","Data":"15664ab368765012c1c0a3e0c01f7359ee27643bb1d53062454cfe9477666790"}
Sep 16 14:43:43.402124 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:43.402088    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s" event={"ID":"7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99","Type":"ContainerStarted","Data":"dd151a8ef22b55ab5fcf8ea847eb16f5a68381d57c49324983e151648ccc0e24"}
Sep 16 14:43:43.421420 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:43:43.421370    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qxh5s" podStartSLOduration=1.84163231 podStartE2EDuration="6.421355615s" totalImagesPullingTime="4.579723305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:43:43.419975863 +0000 UTC m=+500.574054161" watchObservedRunningTime="2026-09-16 14:43:43.421355615 +0000 UTC m=+500.575433902"
Sep 16 14:44:33.818306 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:33.818230    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v9sq5"]
Sep 16 14:44:33.821404 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:33.821389    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:33.823981 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:33.823961    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 14:44:33.829834 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:33.829806    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v9sq5"]
Sep 16 14:44:33.836229 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:33.836203    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkk5b\" (UniqueName: \"kubernetes.io/projected/6064b0c3-514c-482f-afdf-36bcfe770be6-kube-api-access-mkk5b\") pod \"limitador-limitador-8588749977-v9sq5\" (UID: \"6064b0c3-514c-482f-afdf-36bcfe770be6\") " pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:33.836313 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:33.836264    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6064b0c3-514c-482f-afdf-36bcfe770be6-config-file\") pod \"limitador-limitador-8588749977-v9sq5\" (UID: \"6064b0c3-514c-482f-afdf-36bcfe770be6\") " pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:33.911227 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:33.911201    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v9sq5"]
Sep 16 14:44:33.937551 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:33.937525    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkk5b\" (UniqueName: \"kubernetes.io/projected/6064b0c3-514c-482f-afdf-36bcfe770be6-kube-api-access-mkk5b\") pod \"limitador-limitador-8588749977-v9sq5\" (UID: \"6064b0c3-514c-482f-afdf-36bcfe770be6\") " pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:33.937711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:33.937562    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6064b0c3-514c-482f-afdf-36bcfe770be6-config-file\") pod \"limitador-limitador-8588749977-v9sq5\" (UID: \"6064b0c3-514c-482f-afdf-36bcfe770be6\") " pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:33.938092 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:33.938074    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6064b0c3-514c-482f-afdf-36bcfe770be6-config-file\") pod \"limitador-limitador-8588749977-v9sq5\" (UID: \"6064b0c3-514c-482f-afdf-36bcfe770be6\") " pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:33.945563 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:33.945542    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkk5b\" (UniqueName: \"kubernetes.io/projected/6064b0c3-514c-482f-afdf-36bcfe770be6-kube-api-access-mkk5b\") pod \"limitador-limitador-8588749977-v9sq5\" (UID: \"6064b0c3-514c-482f-afdf-36bcfe770be6\") " pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:34.132183 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.132081    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:34.399945 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.399922    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v9sq5"]
Sep 16 14:44:34.402451 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:44:34.402422    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6064b0c3_514c_482f_afdf_36bcfe770be6.slice/crio-fa74cd31f9d66ee39c64f888cd12ebcc6b99b1363e5fde5d4938ab22cf935161 WatchSource:0}: Error finding container fa74cd31f9d66ee39c64f888cd12ebcc6b99b1363e5fde5d4938ab22cf935161: Status 404 returned error can't find the container with id fa74cd31f9d66ee39c64f888cd12ebcc6b99b1363e5fde5d4938ab22cf935161
Sep 16 14:44:34.622648 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.622620    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5" event={"ID":"6064b0c3-514c-482f-afdf-36bcfe770be6","Type":"ContainerStarted","Data":"fa74cd31f9d66ee39c64f888cd12ebcc6b99b1363e5fde5d4938ab22cf935161"}
Sep 16 14:44:34.653547 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.653483    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h7mh7"]
Sep 16 14:44:34.657916 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.657900    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7"
Sep 16 14:44:34.660469 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.660453    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dk6dx\""
Sep 16 14:44:34.664407 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.664382    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h7mh7"]
Sep 16 14:44:34.747026 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.746990    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z4vx\" (UniqueName: \"kubernetes.io/projected/4376b89c-1188-43bb-927c-9b57bca6f8dc-kube-api-access-8z4vx\") pod \"authorino-7bcfd8f9cd-h7mh7\" (UID: \"4376b89c-1188-43bb-927c-9b57bca6f8dc\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7"
Sep 16 14:44:34.794451 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.794432    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-7mgs8"]
Sep 16 14:44:34.797393 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.797378    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7mgs8"
Sep 16 14:44:34.803728 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.803708    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7mgs8"]
Sep 16 14:44:34.847559 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.847528    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6lgv\" (UniqueName: \"kubernetes.io/projected/19a58ba4-1ffa-45e7-b8ec-b422aa874387-kube-api-access-h6lgv\") pod \"authorino-59c974bd48-7mgs8\" (UID: \"19a58ba4-1ffa-45e7-b8ec-b422aa874387\") " pod="kuadrant-system/authorino-59c974bd48-7mgs8"
Sep 16 14:44:34.847843 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.847584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z4vx\" (UniqueName: \"kubernetes.io/projected/4376b89c-1188-43bb-927c-9b57bca6f8dc-kube-api-access-8z4vx\") pod \"authorino-7bcfd8f9cd-h7mh7\" (UID: \"4376b89c-1188-43bb-927c-9b57bca6f8dc\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7"
Sep 16 14:44:34.855605 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.855588    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z4vx\" (UniqueName: \"kubernetes.io/projected/4376b89c-1188-43bb-927c-9b57bca6f8dc-kube-api-access-8z4vx\") pod \"authorino-7bcfd8f9cd-h7mh7\" (UID: \"4376b89c-1188-43bb-927c-9b57bca6f8dc\") " pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7"
Sep 16 14:44:34.948254 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.948208    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6lgv\" (UniqueName: \"kubernetes.io/projected/19a58ba4-1ffa-45e7-b8ec-b422aa874387-kube-api-access-h6lgv\") pod \"authorino-59c974bd48-7mgs8\" (UID: \"19a58ba4-1ffa-45e7-b8ec-b422aa874387\") " pod="kuadrant-system/authorino-59c974bd48-7mgs8"
Sep 16 14:44:34.955945 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.955922    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6lgv\" (UniqueName: \"kubernetes.io/projected/19a58ba4-1ffa-45e7-b8ec-b422aa874387-kube-api-access-h6lgv\") pod \"authorino-59c974bd48-7mgs8\" (UID: \"19a58ba4-1ffa-45e7-b8ec-b422aa874387\") " pod="kuadrant-system/authorino-59c974bd48-7mgs8"
Sep 16 14:44:34.967738 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:34.967719    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7"
Sep 16 14:44:35.105617 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:35.105589    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7mgs8"
Sep 16 14:44:35.233707 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:35.233500    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h7mh7"]
Sep 16 14:44:35.235698 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:44:35.235665    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4376b89c_1188_43bb_927c_9b57bca6f8dc.slice/crio-0e25b1e90883b2eb050403d15a383ce80a4f32147ae734cfa9585446d4dd087d WatchSource:0}: Error finding container 0e25b1e90883b2eb050403d15a383ce80a4f32147ae734cfa9585446d4dd087d: Status 404 returned error can't find the container with id 0e25b1e90883b2eb050403d15a383ce80a4f32147ae734cfa9585446d4dd087d
Sep 16 14:44:35.372416 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:35.372393    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7mgs8"]
Sep 16 14:44:35.375052 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:44:35.375025    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19a58ba4_1ffa_45e7_b8ec_b422aa874387.slice/crio-f2d52da1912bc2d2838425a1aaaa94d71e23e831e74d2cf5c4bd24c06d8501f5 WatchSource:0}: Error finding container f2d52da1912bc2d2838425a1aaaa94d71e23e831e74d2cf5c4bd24c06d8501f5: Status 404 returned error can't find the container with id f2d52da1912bc2d2838425a1aaaa94d71e23e831e74d2cf5c4bd24c06d8501f5
Sep 16 14:44:35.628721 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:35.628682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7mgs8" event={"ID":"19a58ba4-1ffa-45e7-b8ec-b422aa874387","Type":"ContainerStarted","Data":"f2d52da1912bc2d2838425a1aaaa94d71e23e831e74d2cf5c4bd24c06d8501f5"}
Sep 16 14:44:35.630028 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:35.629992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7" event={"ID":"4376b89c-1188-43bb-927c-9b57bca6f8dc","Type":"ContainerStarted","Data":"0e25b1e90883b2eb050403d15a383ce80a4f32147ae734cfa9585446d4dd087d"}
Sep 16 14:44:37.643408 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:37.643373    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5" event={"ID":"6064b0c3-514c-482f-afdf-36bcfe770be6","Type":"ContainerStarted","Data":"b2b0ed4bf563ffc2ce970f6c939308da7533be97faae70f2193069fa6a33bb96"}
Sep 16 14:44:37.643930 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:37.643501    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:37.661316 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:37.661270    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5" podStartSLOduration=1.880677206 podStartE2EDuration="4.661253903s" totalImagesPullingTime="2.780576697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:44:37.659571121 +0000 UTC m=+554.813649410" watchObservedRunningTime="2026-09-16 14:44:37.661253903 +0000 UTC m=+554.815332246"
Sep 16 14:44:39.653871 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:39.653833    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7mgs8" event={"ID":"19a58ba4-1ffa-45e7-b8ec-b422aa874387","Type":"ContainerStarted","Data":"d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b"}
Sep 16 14:44:39.672859 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:39.672807    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-7mgs8" podStartSLOduration=1.754300367 podStartE2EDuration="5.672791911s" totalImagesPullingTime="3.918491544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:44:39.6714984 +0000 UTC m=+556.825576686" watchObservedRunningTime="2026-09-16 14:44:39.672791911 +0000 UTC m=+556.826870216"
Sep 16 14:44:39.696137 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:39.696103    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h7mh7"]
Sep 16 14:44:40.659852 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:40.659817    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7" event={"ID":"4376b89c-1188-43bb-927c-9b57bca6f8dc","Type":"ContainerStarted","Data":"64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d"}
Sep 16 14:44:40.660307 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:40.659852    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7" podUID="4376b89c-1188-43bb-927c-9b57bca6f8dc" containerName="authorino" containerID="cri-o://64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d" gracePeriod=30
Sep 16 14:44:40.677058 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:40.677012    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7" podStartSLOduration=1.7471428709999999 podStartE2EDuration="6.67699312s" totalImagesPullingTime="4.929850249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:44:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:44:40.676370436 +0000 UTC m=+557.830448754" watchObservedRunningTime="2026-09-16 14:44:40.67699312 +0000 UTC m=+557.831071409"
Sep 16 14:44:40.903262 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:40.903240    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7"
Sep 16 14:44:41.002311 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.002248    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z4vx\" (UniqueName: \"kubernetes.io/projected/4376b89c-1188-43bb-927c-9b57bca6f8dc-kube-api-access-8z4vx\") pod \"4376b89c-1188-43bb-927c-9b57bca6f8dc\" (UID: \"4376b89c-1188-43bb-927c-9b57bca6f8dc\") "
Sep 16 14:44:41.004538 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.004511    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4376b89c-1188-43bb-927c-9b57bca6f8dc-kube-api-access-8z4vx" (OuterVolumeSpecName: "kube-api-access-8z4vx") pod "4376b89c-1188-43bb-927c-9b57bca6f8dc" (UID: "4376b89c-1188-43bb-927c-9b57bca6f8dc"). InnerVolumeSpecName "kube-api-access-8z4vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:44:41.103157 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.103131    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z4vx\" (UniqueName: \"kubernetes.io/projected/4376b89c-1188-43bb-927c-9b57bca6f8dc-kube-api-access-8z4vx\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:44:41.665704 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.665667    2558 generic.go:356] "Generic (PLEG): container finished" podID="4376b89c-1188-43bb-927c-9b57bca6f8dc" containerID="64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d" exitCode=0
Sep 16 14:44:41.665704 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.665704    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7"
Sep 16 14:44:41.666131 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.665708    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7" event={"ID":"4376b89c-1188-43bb-927c-9b57bca6f8dc","Type":"ContainerDied","Data":"64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d"}
Sep 16 14:44:41.666131 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.665802    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-h7mh7" event={"ID":"4376b89c-1188-43bb-927c-9b57bca6f8dc","Type":"ContainerDied","Data":"0e25b1e90883b2eb050403d15a383ce80a4f32147ae734cfa9585446d4dd087d"}
Sep 16 14:44:41.666131 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.665819    2558 scope.go:117] "RemoveContainer" containerID="64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d"
Sep 16 14:44:41.673608 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.673592    2558 scope.go:117] "RemoveContainer" containerID="64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d"
Sep 16 14:44:41.673857 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:44:41.673842    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d\": container with ID starting with 64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d not found: ID does not exist" containerID="64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d"
Sep 16 14:44:41.673902 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.673866    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d"} err="failed to get container status \"64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d\": rpc error: code = NotFound desc = could not find container \"64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d\": container with ID starting with 64a7ed0ce853a4697c44e69c414204cfd123fe856c6bb2f18a00d1b156e4051d not found: ID does not exist"
Sep 16 14:44:41.681690 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.681668    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h7mh7"]
Sep 16 14:44:41.685487 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:41.685460    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-h7mh7"]
Sep 16 14:44:43.294069 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:43.294037    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4376b89c-1188-43bb-927c-9b57bca6f8dc" path="/var/lib/kubelet/pods/4376b89c-1188-43bb-927c-9b57bca6f8dc/volumes"
Sep 16 14:44:48.649804 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:48.649772    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:53.221948 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:53.221919    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v9sq5"]
Sep 16 14:44:53.222430 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:53.222092    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5" podUID="6064b0c3-514c-482f-afdf-36bcfe770be6" containerName="limitador" containerID="cri-o://b2b0ed4bf563ffc2ce970f6c939308da7533be97faae70f2193069fa6a33bb96" gracePeriod=30
Sep 16 14:44:53.719123 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:53.719094    2558 generic.go:356] "Generic (PLEG): container finished" podID="6064b0c3-514c-482f-afdf-36bcfe770be6" containerID="b2b0ed4bf563ffc2ce970f6c939308da7533be97faae70f2193069fa6a33bb96" exitCode=0
Sep 16 14:44:53.719272 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:53.719155    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5" event={"ID":"6064b0c3-514c-482f-afdf-36bcfe770be6","Type":"ContainerDied","Data":"b2b0ed4bf563ffc2ce970f6c939308da7533be97faae70f2193069fa6a33bb96"}
Sep 16 14:44:53.759283 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:53.759263    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:53.900240 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:53.900210    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkk5b\" (UniqueName: \"kubernetes.io/projected/6064b0c3-514c-482f-afdf-36bcfe770be6-kube-api-access-mkk5b\") pod \"6064b0c3-514c-482f-afdf-36bcfe770be6\" (UID: \"6064b0c3-514c-482f-afdf-36bcfe770be6\") "
Sep 16 14:44:53.900384 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:53.900250    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6064b0c3-514c-482f-afdf-36bcfe770be6-config-file\") pod \"6064b0c3-514c-482f-afdf-36bcfe770be6\" (UID: \"6064b0c3-514c-482f-afdf-36bcfe770be6\") "
Sep 16 14:44:53.900626 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:53.900602    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6064b0c3-514c-482f-afdf-36bcfe770be6-config-file" (OuterVolumeSpecName: "config-file") pod "6064b0c3-514c-482f-afdf-36bcfe770be6" (UID: "6064b0c3-514c-482f-afdf-36bcfe770be6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:44:53.902249 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:53.902225    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6064b0c3-514c-482f-afdf-36bcfe770be6-kube-api-access-mkk5b" (OuterVolumeSpecName: "kube-api-access-mkk5b") pod "6064b0c3-514c-482f-afdf-36bcfe770be6" (UID: "6064b0c3-514c-482f-afdf-36bcfe770be6"). InnerVolumeSpecName "kube-api-access-mkk5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:44:54.001715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:54.001682    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkk5b\" (UniqueName: \"kubernetes.io/projected/6064b0c3-514c-482f-afdf-36bcfe770be6-kube-api-access-mkk5b\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:44:54.001715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:54.001714    2558 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6064b0c3-514c-482f-afdf-36bcfe770be6-config-file\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:44:54.725064 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:54.725032    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5"
Sep 16 14:44:54.725064 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:54.725043    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v9sq5" event={"ID":"6064b0c3-514c-482f-afdf-36bcfe770be6","Type":"ContainerDied","Data":"fa74cd31f9d66ee39c64f888cd12ebcc6b99b1363e5fde5d4938ab22cf935161"}
Sep 16 14:44:54.725531 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:54.725094    2558 scope.go:117] "RemoveContainer" containerID="b2b0ed4bf563ffc2ce970f6c939308da7533be97faae70f2193069fa6a33bb96"
Sep 16 14:44:54.745925 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:54.745903    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v9sq5"]
Sep 16 14:44:54.747731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:54.747706    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v9sq5"]
Sep 16 14:44:55.294303 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:55.294270    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6064b0c3-514c-482f-afdf-36bcfe770be6" path="/var/lib/kubelet/pods/6064b0c3-514c-482f-afdf-36bcfe770be6/volumes"
Sep 16 14:44:57.041287 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.041256    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"]
Sep 16 14:44:57.041683 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.041565    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6064b0c3-514c-482f-afdf-36bcfe770be6" containerName="limitador"
Sep 16 14:44:57.041683 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.041576    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6064b0c3-514c-482f-afdf-36bcfe770be6" containerName="limitador"
Sep 16 14:44:57.041683 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.041583    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4376b89c-1188-43bb-927c-9b57bca6f8dc" containerName="authorino"
Sep 16 14:44:57.041683 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.041589    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4376b89c-1188-43bb-927c-9b57bca6f8dc" containerName="authorino"
Sep 16 14:44:57.041683 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.041639    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4376b89c-1188-43bb-927c-9b57bca6f8dc" containerName="authorino"
Sep 16 14:44:57.041683 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.041648    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6064b0c3-514c-482f-afdf-36bcfe770be6" containerName="limitador"
Sep 16 14:44:57.045800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.045778    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"
Sep 16 14:44:57.048772 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.048755    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 14:44:57.048885 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.048759    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 16 14:44:57.049593 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.049573    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-v2vgv\""
Sep 16 14:44:57.049683 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.049575    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 14:44:57.053529 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.053510    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"]
Sep 16 14:44:57.124889 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.124865    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c-data\") pod \"postgres-69fb4594cf-jcfz4\" (UID: \"c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"
Sep 16 14:44:57.124983 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.124925    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m494c\" (UniqueName: \"kubernetes.io/projected/c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c-kube-api-access-m494c\") pod \"postgres-69fb4594cf-jcfz4\" (UID: \"c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"
Sep 16 14:44:57.226330 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.226305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m494c\" (UniqueName: \"kubernetes.io/projected/c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c-kube-api-access-m494c\") pod \"postgres-69fb4594cf-jcfz4\" (UID: \"c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"
Sep 16 14:44:57.226421 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.226348    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c-data\") pod \"postgres-69fb4594cf-jcfz4\" (UID: \"c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"
Sep 16 14:44:57.226665 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.226650    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c-data\") pod \"postgres-69fb4594cf-jcfz4\" (UID: \"c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"
Sep 16 14:44:57.235242 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.235217    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m494c\" (UniqueName: \"kubernetes.io/projected/c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c-kube-api-access-m494c\") pod \"postgres-69fb4594cf-jcfz4\" (UID: \"c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"
Sep 16 14:44:57.356669 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.356632    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"
Sep 16 14:44:57.626002 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.625978    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"]
Sep 16 14:44:57.628384 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:44:57.628355    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cfbe47_6d5b_420e_abb0_c3cc59a6b69c.slice/crio-fc5659f564083e868a8ab0b2c8a6b22157419131d6155133b6e6fda7fc67b4e9 WatchSource:0}: Error finding container fc5659f564083e868a8ab0b2c8a6b22157419131d6155133b6e6fda7fc67b4e9: Status 404 returned error can't find the container with id fc5659f564083e868a8ab0b2c8a6b22157419131d6155133b6e6fda7fc67b4e9
Sep 16 14:44:57.739892 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:44:57.739860    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4" event={"ID":"c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c","Type":"ContainerStarted","Data":"fc5659f564083e868a8ab0b2c8a6b22157419131d6155133b6e6fda7fc67b4e9"}
Sep 16 14:45:02.768780 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:02.768684    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4" event={"ID":"c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c","Type":"ContainerStarted","Data":"76ebc64dd5c0ed7a80e87dc5c889d34a124c1d5c3567c646ddcfcfa851a7abaf"}
Sep 16 14:45:02.768780 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:02.768767    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"
Sep 16 14:45:02.785676 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:02.785635    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4" podStartSLOduration=1.105605227 podStartE2EDuration="5.785622872s" totalImagesPullingTime="4.680017645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:02.784999336 +0000 UTC m=+579.939077624" watchObservedRunningTime="2026-09-16 14:45:02.785622872 +0000 UTC m=+579.939701158"
Sep 16 14:45:08.800355 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:08.800322    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jcfz4"
Sep 16 14:45:09.400988 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.400956    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-6chxf"]
Sep 16 14:45:09.407370 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.407353    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6chxf"
Sep 16 14:45:09.413059 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.413026    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-6chxf"]
Sep 16 14:45:09.528969 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.528929    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7pr2\" (UniqueName: \"kubernetes.io/projected/209f05d9-a6d0-48f2-8f3c-ca1cdee004dd-kube-api-access-q7pr2\") pod \"authorino-557f876f69-6chxf\" (UID: \"209f05d9-a6d0-48f2-8f3c-ca1cdee004dd\") " pod="kuadrant-system/authorino-557f876f69-6chxf"
Sep 16 14:45:09.629365 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.629323    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7pr2\" (UniqueName: \"kubernetes.io/projected/209f05d9-a6d0-48f2-8f3c-ca1cdee004dd-kube-api-access-q7pr2\") pod \"authorino-557f876f69-6chxf\" (UID: \"209f05d9-a6d0-48f2-8f3c-ca1cdee004dd\") " pod="kuadrant-system/authorino-557f876f69-6chxf"
Sep 16 14:45:09.638686 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.638658    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7pr2\" (UniqueName: \"kubernetes.io/projected/209f05d9-a6d0-48f2-8f3c-ca1cdee004dd-kube-api-access-q7pr2\") pod \"authorino-557f876f69-6chxf\" (UID: \"209f05d9-a6d0-48f2-8f3c-ca1cdee004dd\") " pod="kuadrant-system/authorino-557f876f69-6chxf"
Sep 16 14:45:09.696103 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.696037    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6chxf"]
Sep 16 14:45:09.696299 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.696286    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6chxf"
Sep 16 14:45:09.740678 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.740648    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-769dddbcc5-frw52"]
Sep 16 14:45:09.746471 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.746256    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769dddbcc5-frw52"
Sep 16 14:45:09.762879 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.762815    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-769dddbcc5-frw52"]
Sep 16 14:45:09.831495 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.831472    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx2z4\" (UniqueName: \"kubernetes.io/projected/e439bc2f-e646-4a6b-a3a7-bc24a3a17cba-kube-api-access-dx2z4\") pod \"authorino-769dddbcc5-frw52\" (UID: \"e439bc2f-e646-4a6b-a3a7-bc24a3a17cba\") " pod="kuadrant-system/authorino-769dddbcc5-frw52"
Sep 16 14:45:09.932818 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.932792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx2z4\" (UniqueName: \"kubernetes.io/projected/e439bc2f-e646-4a6b-a3a7-bc24a3a17cba-kube-api-access-dx2z4\") pod \"authorino-769dddbcc5-frw52\" (UID: \"e439bc2f-e646-4a6b-a3a7-bc24a3a17cba\") " pod="kuadrant-system/authorino-769dddbcc5-frw52"
Sep 16 14:45:09.947914 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.947857    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx2z4\" (UniqueName: \"kubernetes.io/projected/e439bc2f-e646-4a6b-a3a7-bc24a3a17cba-kube-api-access-dx2z4\") pod \"authorino-769dddbcc5-frw52\" (UID: \"e439bc2f-e646-4a6b-a3a7-bc24a3a17cba\") " pod="kuadrant-system/authorino-769dddbcc5-frw52"
Sep 16 14:45:09.981330 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:09.981304    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6chxf"]
Sep 16 14:45:09.984280 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:45:09.984256    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod209f05d9_a6d0_48f2_8f3c_ca1cdee004dd.slice/crio-69e24143ec0530fb9dab0481138162359cb4c79c5d621cd4bb5b4ba97b0798b8 WatchSource:0}: Error finding container 69e24143ec0530fb9dab0481138162359cb4c79c5d621cd4bb5b4ba97b0798b8: Status 404 returned error can't find the container with id 69e24143ec0530fb9dab0481138162359cb4c79c5d621cd4bb5b4ba97b0798b8
Sep 16 14:45:10.059893 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:10.059869    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769dddbcc5-frw52"
Sep 16 14:45:10.328151 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:10.328130    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-769dddbcc5-frw52"]
Sep 16 14:45:10.330468 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:45:10.330443    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode439bc2f_e646_4a6b_a3a7_bc24a3a17cba.slice/crio-07917f7eee83912df76e57b4ce070961eb28837dd1d13953525516d841de3e82 WatchSource:0}: Error finding container 07917f7eee83912df76e57b4ce070961eb28837dd1d13953525516d841de3e82: Status 404 returned error can't find the container with id 07917f7eee83912df76e57b4ce070961eb28837dd1d13953525516d841de3e82
Sep 16 14:45:10.804872 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:10.804833    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-769dddbcc5-frw52" event={"ID":"e439bc2f-e646-4a6b-a3a7-bc24a3a17cba","Type":"ContainerStarted","Data":"07917f7eee83912df76e57b4ce070961eb28837dd1d13953525516d841de3e82"}
Sep 16 14:45:10.807087 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:10.807053    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-6chxf" podUID="209f05d9-a6d0-48f2-8f3c-ca1cdee004dd" containerName="authorino" containerID="cri-o://12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5" gracePeriod=30
Sep 16 14:45:10.807235 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:10.806905    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-6chxf" event={"ID":"209f05d9-a6d0-48f2-8f3c-ca1cdee004dd","Type":"ContainerStarted","Data":"12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5"}
Sep 16 14:45:10.807307 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:10.807244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-6chxf" event={"ID":"209f05d9-a6d0-48f2-8f3c-ca1cdee004dd","Type":"ContainerStarted","Data":"69e24143ec0530fb9dab0481138162359cb4c79c5d621cd4bb5b4ba97b0798b8"}
Sep 16 14:45:10.825316 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:10.825275    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-6chxf" podStartSLOduration=1.201232189 podStartE2EDuration="1.825261062s" totalImagesPullingTime="624.028873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:10.823237685 +0000 UTC m=+587.977315969" watchObservedRunningTime="2026-09-16 14:45:10.825261062 +0000 UTC m=+587.979339349"
Sep 16 14:45:11.077596 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.077572    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6chxf"
Sep 16 14:45:11.243816 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.243737    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7pr2\" (UniqueName: \"kubernetes.io/projected/209f05d9-a6d0-48f2-8f3c-ca1cdee004dd-kube-api-access-q7pr2\") pod \"209f05d9-a6d0-48f2-8f3c-ca1cdee004dd\" (UID: \"209f05d9-a6d0-48f2-8f3c-ca1cdee004dd\") "
Sep 16 14:45:11.245699 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.245675    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209f05d9-a6d0-48f2-8f3c-ca1cdee004dd-kube-api-access-q7pr2" (OuterVolumeSpecName: "kube-api-access-q7pr2") pod "209f05d9-a6d0-48f2-8f3c-ca1cdee004dd" (UID: "209f05d9-a6d0-48f2-8f3c-ca1cdee004dd"). InnerVolumeSpecName "kube-api-access-q7pr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:45:11.344884 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.344850    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7pr2\" (UniqueName: \"kubernetes.io/projected/209f05d9-a6d0-48f2-8f3c-ca1cdee004dd-kube-api-access-q7pr2\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:45:11.812723 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.812687    2558 generic.go:356] "Generic (PLEG): container finished" podID="209f05d9-a6d0-48f2-8f3c-ca1cdee004dd" containerID="12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5" exitCode=0
Sep 16 14:45:11.812723 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.812728    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-6chxf"
Sep 16 14:45:11.812953 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.812771    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-6chxf" event={"ID":"209f05d9-a6d0-48f2-8f3c-ca1cdee004dd","Type":"ContainerDied","Data":"12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5"}
Sep 16 14:45:11.812953 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.812807    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-6chxf" event={"ID":"209f05d9-a6d0-48f2-8f3c-ca1cdee004dd","Type":"ContainerDied","Data":"69e24143ec0530fb9dab0481138162359cb4c79c5d621cd4bb5b4ba97b0798b8"}
Sep 16 14:45:11.812953 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.812829    2558 scope.go:117] "RemoveContainer" containerID="12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5"
Sep 16 14:45:11.814444 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.814419    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-769dddbcc5-frw52" event={"ID":"e439bc2f-e646-4a6b-a3a7-bc24a3a17cba","Type":"ContainerStarted","Data":"d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3"}
Sep 16 14:45:11.820827 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.820814    2558 scope.go:117] "RemoveContainer" containerID="12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5"
Sep 16 14:45:11.821085 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:45:11.821067    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5\": container with ID starting with 12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5 not found: ID does not exist" containerID="12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5"
Sep 16 14:45:11.821135 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.821094    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5"} err="failed to get container status \"12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5\": rpc error: code = NotFound desc = could not find container \"12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5\": container with ID starting with 12fe88e1500abb40032868a083b4641e49812dc39e9a21dd5ba50cbd94fbf3f5 not found: ID does not exist"
Sep 16 14:45:11.827819 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.827795    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-6chxf"]
Sep 16 14:45:11.833774 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.833756    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-6chxf"]
Sep 16 14:45:11.846623 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.846584    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-769dddbcc5-frw52" podStartSLOduration=2.12770366 podStartE2EDuration="2.846571736s" totalImagesPullingTime="718.868076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:11.845360031 +0000 UTC m=+588.999438327" watchObservedRunningTime="2026-09-16 14:45:11.846571736 +0000 UTC m=+589.000650055"
Sep 16 14:45:11.872471 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.872434    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7mgs8"]
Sep 16 14:45:11.872633 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.872606    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-7mgs8" podUID="19a58ba4-1ffa-45e7-b8ec-b422aa874387" containerName="authorino" containerID="cri-o://d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b" gracePeriod=30
Sep 16 14:45:11.950008 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.949939    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f9cd9c497-xxzxq"]
Sep 16 14:45:11.950325 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.950311    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="209f05d9-a6d0-48f2-8f3c-ca1cdee004dd" containerName="authorino"
Sep 16 14:45:11.950377 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.950328    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="209f05d9-a6d0-48f2-8f3c-ca1cdee004dd" containerName="authorino"
Sep 16 14:45:11.950425 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.950384    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="209f05d9-a6d0-48f2-8f3c-ca1cdee004dd" containerName="authorino"
Sep 16 14:45:11.966492 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.966470    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f9cd9c497-xxzxq"]
Sep 16 14:45:11.966605 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.966473    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f9cd9c497-xxzxq"
Sep 16 14:45:11.970204 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:11.970186    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 14:45:12.051925 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.051883    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-tls-cert\") pod \"authorino-7f9cd9c497-xxzxq\" (UID: \"6552d805-211a-46ab-8f7f-0b5dbc1b5cec\") " pod="kuadrant-system/authorino-7f9cd9c497-xxzxq"
Sep 16 14:45:12.051925 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.051922    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n4f8\" (UniqueName: \"kubernetes.io/projected/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-kube-api-access-6n4f8\") pod \"authorino-7f9cd9c497-xxzxq\" (UID: \"6552d805-211a-46ab-8f7f-0b5dbc1b5cec\") " pod="kuadrant-system/authorino-7f9cd9c497-xxzxq"
Sep 16 14:45:12.152847 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.152820    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-tls-cert\") pod \"authorino-7f9cd9c497-xxzxq\" (UID: \"6552d805-211a-46ab-8f7f-0b5dbc1b5cec\") " pod="kuadrant-system/authorino-7f9cd9c497-xxzxq"
Sep 16 14:45:12.153208 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.152860    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n4f8\" (UniqueName: \"kubernetes.io/projected/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-kube-api-access-6n4f8\") pod \"authorino-7f9cd9c497-xxzxq\" (UID: \"6552d805-211a-46ab-8f7f-0b5dbc1b5cec\") " pod="kuadrant-system/authorino-7f9cd9c497-xxzxq"
Sep 16 14:45:12.155238 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.155217    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-tls-cert\") pod \"authorino-7f9cd9c497-xxzxq\" (UID: \"6552d805-211a-46ab-8f7f-0b5dbc1b5cec\") " pod="kuadrant-system/authorino-7f9cd9c497-xxzxq"
Sep 16 14:45:12.159453 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.159434    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7mgs8"
Sep 16 14:45:12.161193 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.161149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n4f8\" (UniqueName: \"kubernetes.io/projected/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-kube-api-access-6n4f8\") pod \"authorino-7f9cd9c497-xxzxq\" (UID: \"6552d805-211a-46ab-8f7f-0b5dbc1b5cec\") " pod="kuadrant-system/authorino-7f9cd9c497-xxzxq"
Sep 16 14:45:12.254198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.254115    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6lgv\" (UniqueName: \"kubernetes.io/projected/19a58ba4-1ffa-45e7-b8ec-b422aa874387-kube-api-access-h6lgv\") pod \"19a58ba4-1ffa-45e7-b8ec-b422aa874387\" (UID: \"19a58ba4-1ffa-45e7-b8ec-b422aa874387\") "
Sep 16 14:45:12.256013 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.255990    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a58ba4-1ffa-45e7-b8ec-b422aa874387-kube-api-access-h6lgv" (OuterVolumeSpecName: "kube-api-access-h6lgv") pod "19a58ba4-1ffa-45e7-b8ec-b422aa874387" (UID: "19a58ba4-1ffa-45e7-b8ec-b422aa874387"). InnerVolumeSpecName "kube-api-access-h6lgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:45:12.308425 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.308405    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f9cd9c497-xxzxq"
Sep 16 14:45:12.355223 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.355198    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6lgv\" (UniqueName: \"kubernetes.io/projected/19a58ba4-1ffa-45e7-b8ec-b422aa874387-kube-api-access-h6lgv\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:45:12.596077 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.596055    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f9cd9c497-xxzxq"]
Sep 16 14:45:12.598406 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:45:12.598376    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6552d805_211a_46ab_8f7f_0b5dbc1b5cec.slice/crio-aa69c8dfaa16ffa5e14cc4a41b2b291ee209b7fbd1b64b320dfd91decafba08b WatchSource:0}: Error finding container aa69c8dfaa16ffa5e14cc4a41b2b291ee209b7fbd1b64b320dfd91decafba08b: Status 404 returned error can't find the container with id aa69c8dfaa16ffa5e14cc4a41b2b291ee209b7fbd1b64b320dfd91decafba08b
Sep 16 14:45:12.823610 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.823580    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f9cd9c497-xxzxq" event={"ID":"6552d805-211a-46ab-8f7f-0b5dbc1b5cec","Type":"ContainerStarted","Data":"aa69c8dfaa16ffa5e14cc4a41b2b291ee209b7fbd1b64b320dfd91decafba08b"}
Sep 16 14:45:12.824670 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.824640    2558 generic.go:356] "Generic (PLEG): container finished" podID="19a58ba4-1ffa-45e7-b8ec-b422aa874387" containerID="d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b" exitCode=0
Sep 16 14:45:12.824795 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.824680    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-7mgs8"
Sep 16 14:45:12.824795 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.824719    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7mgs8" event={"ID":"19a58ba4-1ffa-45e7-b8ec-b422aa874387","Type":"ContainerDied","Data":"d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b"}
Sep 16 14:45:12.824795 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.824758    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-7mgs8" event={"ID":"19a58ba4-1ffa-45e7-b8ec-b422aa874387","Type":"ContainerDied","Data":"f2d52da1912bc2d2838425a1aaaa94d71e23e831e74d2cf5c4bd24c06d8501f5"}
Sep 16 14:45:12.824795 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.824776    2558 scope.go:117] "RemoveContainer" containerID="d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b"
Sep 16 14:45:12.832790 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.832771    2558 scope.go:117] "RemoveContainer" containerID="d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b"
Sep 16 14:45:12.833042 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:45:12.833020    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b\": container with ID starting with d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b not found: ID does not exist" containerID="d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b"
Sep 16 14:45:12.833127 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.833050    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b"} err="failed to get container status \"d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b\": rpc error: code = NotFound desc = could not find container \"d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b\": container with ID starting with d3ce38c780073b1c00283f1cfde2952c0d54e01bdccd06aa43152539a060dd7b not found: ID does not exist"
Sep 16 14:45:12.845134 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.845114    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7mgs8"]
Sep 16 14:45:12.851270 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:12.851220    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-7mgs8"]
Sep 16 14:45:13.294912 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:13.294880    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19a58ba4-1ffa-45e7-b8ec-b422aa874387" path="/var/lib/kubelet/pods/19a58ba4-1ffa-45e7-b8ec-b422aa874387/volumes"
Sep 16 14:45:13.295298 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:13.295207    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="209f05d9-a6d0-48f2-8f3c-ca1cdee004dd" path="/var/lib/kubelet/pods/209f05d9-a6d0-48f2-8f3c-ca1cdee004dd/volumes"
Sep 16 14:45:13.831561 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:13.831521    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f9cd9c497-xxzxq" event={"ID":"6552d805-211a-46ab-8f7f-0b5dbc1b5cec","Type":"ContainerStarted","Data":"6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893"}
Sep 16 14:45:13.847762 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:13.847711    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f9cd9c497-xxzxq" podStartSLOduration=2.358361927 podStartE2EDuration="2.847697215s" totalImagesPullingTime="489.335288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:13.84750488 +0000 UTC m=+591.001583166" watchObservedRunningTime="2026-09-16 14:45:13.847697215 +0000 UTC m=+591.001775502"
Sep 16 14:45:13.872246 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:13.872213    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-769dddbcc5-frw52"]
Sep 16 14:45:13.872411 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:13.872392    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-769dddbcc5-frw52" podUID="e439bc2f-e646-4a6b-a3a7-bc24a3a17cba" containerName="authorino" containerID="cri-o://d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3" gracePeriod=30
Sep 16 14:45:14.105134 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.105112    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769dddbcc5-frw52"
Sep 16 14:45:14.271752 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.271713    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx2z4\" (UniqueName: \"kubernetes.io/projected/e439bc2f-e646-4a6b-a3a7-bc24a3a17cba-kube-api-access-dx2z4\") pod \"e439bc2f-e646-4a6b-a3a7-bc24a3a17cba\" (UID: \"e439bc2f-e646-4a6b-a3a7-bc24a3a17cba\") "
Sep 16 14:45:14.273474 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.273448    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e439bc2f-e646-4a6b-a3a7-bc24a3a17cba-kube-api-access-dx2z4" (OuterVolumeSpecName: "kube-api-access-dx2z4") pod "e439bc2f-e646-4a6b-a3a7-bc24a3a17cba" (UID: "e439bc2f-e646-4a6b-a3a7-bc24a3a17cba"). InnerVolumeSpecName "kube-api-access-dx2z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:45:14.372963 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.372893    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx2z4\" (UniqueName: \"kubernetes.io/projected/e439bc2f-e646-4a6b-a3a7-bc24a3a17cba-kube-api-access-dx2z4\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:45:14.838288 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.838245    2558 generic.go:356] "Generic (PLEG): container finished" podID="e439bc2f-e646-4a6b-a3a7-bc24a3a17cba" containerID="d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3" exitCode=0
Sep 16 14:45:14.838288 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.838281    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-769dddbcc5-frw52"
Sep 16 14:45:14.838460 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.838331    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-769dddbcc5-frw52" event={"ID":"e439bc2f-e646-4a6b-a3a7-bc24a3a17cba","Type":"ContainerDied","Data":"d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3"}
Sep 16 14:45:14.838460 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.838365    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-769dddbcc5-frw52" event={"ID":"e439bc2f-e646-4a6b-a3a7-bc24a3a17cba","Type":"ContainerDied","Data":"07917f7eee83912df76e57b4ce070961eb28837dd1d13953525516d841de3e82"}
Sep 16 14:45:14.838460 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.838381    2558 scope.go:117] "RemoveContainer" containerID="d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3"
Sep 16 14:45:14.846492 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.846475    2558 scope.go:117] "RemoveContainer" containerID="d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3"
Sep 16 14:45:14.846748 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:45:14.846729    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3\": container with ID starting with d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3 not found: ID does not exist" containerID="d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3"
Sep 16 14:45:14.846809 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.846755    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3"} err="failed to get container status \"d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3\": rpc error: code = NotFound desc = could not find container \"d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3\": container with ID starting with d555dea5ef26722d69145b87a2b1e3200f4b41816e295c190fee0eaa6d41c0a3 not found: ID does not exist"
Sep 16 14:45:14.858698 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.858678    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-769dddbcc5-frw52"]
Sep 16 14:45:14.862553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:14.862532    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-769dddbcc5-frw52"]
Sep 16 14:45:15.295453 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:15.295381    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e439bc2f-e646-4a6b-a3a7-bc24a3a17cba" path="/var/lib/kubelet/pods/e439bc2f-e646-4a6b-a3a7-bc24a3a17cba/volumes"
Sep 16 14:45:16.997850 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:16.997815    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-kdrh7"]
Sep 16 14:45:16.998322 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:16.998277    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19a58ba4-1ffa-45e7-b8ec-b422aa874387" containerName="authorino"
Sep 16 14:45:16.998395 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:16.998326    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="19a58ba4-1ffa-45e7-b8ec-b422aa874387" containerName="authorino"
Sep 16 14:45:16.998395 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:16.998349    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e439bc2f-e646-4a6b-a3a7-bc24a3a17cba" containerName="authorino"
Sep 16 14:45:16.998395 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:16.998357    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e439bc2f-e646-4a6b-a3a7-bc24a3a17cba" containerName="authorino"
Sep 16 14:45:16.998540 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:16.998457    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="19a58ba4-1ffa-45e7-b8ec-b422aa874387" containerName="authorino"
Sep 16 14:45:16.998540 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:16.998470    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e439bc2f-e646-4a6b-a3a7-bc24a3a17cba" containerName="authorino"
Sep 16 14:45:17.002928 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.002908    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:17.005853 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.005834    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 14:45:17.005943 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.005874    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tcknf\""
Sep 16 14:45:17.006429 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.006415    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 14:45:17.007051 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.007037    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 14:45:17.017364 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.017345    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-kdrh7"]
Sep 16 14:45:17.196900 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.196869    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4gjs\" (UniqueName: \"kubernetes.io/projected/8deea262-2fc4-4f53-ad0f-8174c5adfea1-kube-api-access-t4gjs\") pod \"maas-controller-65cbd6bdf6-kdrh7\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") " pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:17.197063 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.196917    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-metrics-certs\") pod \"maas-controller-65cbd6bdf6-kdrh7\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") " pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:17.197063 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.196940    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-webhook-cert\") pod \"maas-controller-65cbd6bdf6-kdrh7\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") " pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:17.285431 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.285364    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-pdbbb"]
Sep 16 14:45:17.288637 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.288621    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:17.298101 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.298079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4gjs\" (UniqueName: \"kubernetes.io/projected/8deea262-2fc4-4f53-ad0f-8174c5adfea1-kube-api-access-t4gjs\") pod \"maas-controller-65cbd6bdf6-kdrh7\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") " pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:17.298362 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.298335    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-metrics-certs\") pod \"maas-controller-65cbd6bdf6-kdrh7\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") " pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:17.298488 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.298392    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-webhook-cert\") pod \"maas-controller-65cbd6bdf6-kdrh7\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") " pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:17.298569 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:45:17.298530    2558 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 16 14:45:17.298639 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:45:17.298617    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-webhook-cert podName:8deea262-2fc4-4f53-ad0f-8174c5adfea1 nodeName:}" failed. No retries permitted until 2026-09-16 14:45:17.798599799 +0000 UTC m=+594.952678078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-webhook-cert") pod "maas-controller-65cbd6bdf6-kdrh7" (UID: "8deea262-2fc4-4f53-ad0f-8174c5adfea1") : secret "maas-controller-webhook-cert" not found
Sep 16 14:45:17.301288 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.301267    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-metrics-certs\") pod \"maas-controller-65cbd6bdf6-kdrh7\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") " pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:17.309048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.309025    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-pdbbb"]
Sep 16 14:45:17.319153 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.319127    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4gjs\" (UniqueName: \"kubernetes.io/projected/8deea262-2fc4-4f53-ad0f-8174c5adfea1-kube-api-access-t4gjs\") pod \"maas-controller-65cbd6bdf6-kdrh7\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") " pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:17.399688 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.399661    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-metrics-certs\") pod \"maas-controller-5cb4b58ccc-pdbbb\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") " pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:17.399852 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.399697    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcl98\" (UniqueName: \"kubernetes.io/projected/15cab74b-508c-4f12-9e28-20e6adca900f-kube-api-access-hcl98\") pod \"maas-controller-5cb4b58ccc-pdbbb\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") " pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:17.399852 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.399721    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-webhook-cert\") pod \"maas-controller-5cb4b58ccc-pdbbb\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") " pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:17.500403 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.500371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-metrics-certs\") pod \"maas-controller-5cb4b58ccc-pdbbb\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") " pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:17.500403 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.500404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcl98\" (UniqueName: \"kubernetes.io/projected/15cab74b-508c-4f12-9e28-20e6adca900f-kube-api-access-hcl98\") pod \"maas-controller-5cb4b58ccc-pdbbb\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") " pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:17.500626 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.500427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-webhook-cert\") pod \"maas-controller-5cb4b58ccc-pdbbb\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") " pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:17.502771 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.502746    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-webhook-cert\") pod \"maas-controller-5cb4b58ccc-pdbbb\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") " pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:17.503027 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.503004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-metrics-certs\") pod \"maas-controller-5cb4b58ccc-pdbbb\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") " pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:17.515110 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.515088    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcl98\" (UniqueName: \"kubernetes.io/projected/15cab74b-508c-4f12-9e28-20e6adca900f-kube-api-access-hcl98\") pod \"maas-controller-5cb4b58ccc-pdbbb\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") " pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:17.601780 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.601751    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:17.804044 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.803995    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-webhook-cert\") pod \"maas-controller-65cbd6bdf6-kdrh7\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") " pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:17.806300 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.806282    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-webhook-cert\") pod \"maas-controller-65cbd6bdf6-kdrh7\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") " pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:17.872421 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.872399    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-pdbbb"]
Sep 16 14:45:17.874273 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:45:17.874246    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cab74b_508c_4f12_9e28_20e6adca900f.slice/crio-bb02096cfeb5432cf67e4991824438f1e916e732a8d7ff6b5e4f866c9c5c6e28 WatchSource:0}: Error finding container bb02096cfeb5432cf67e4991824438f1e916e732a8d7ff6b5e4f866c9c5c6e28: Status 404 returned error can't find the container with id bb02096cfeb5432cf67e4991824438f1e916e732a8d7ff6b5e4f866c9c5c6e28
Sep 16 14:45:17.913033 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:17.913008    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:18.183056 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:18.183032    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-kdrh7"]
Sep 16 14:45:18.185107 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:45:18.185085    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8deea262_2fc4_4f53_ad0f_8174c5adfea1.slice/crio-f60a638bb540a5bcf93e3a09e0a0513530a808129e5f25101a7f90d1a5703a75 WatchSource:0}: Error finding container f60a638bb540a5bcf93e3a09e0a0513530a808129e5f25101a7f90d1a5703a75: Status 404 returned error can't find the container with id f60a638bb540a5bcf93e3a09e0a0513530a808129e5f25101a7f90d1a5703a75
Sep 16 14:45:18.861914 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:18.861828    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7" event={"ID":"8deea262-2fc4-4f53-ad0f-8174c5adfea1","Type":"ContainerStarted","Data":"f60a638bb540a5bcf93e3a09e0a0513530a808129e5f25101a7f90d1a5703a75"}
Sep 16 14:45:18.864415 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:18.864360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb" event={"ID":"15cab74b-508c-4f12-9e28-20e6adca900f","Type":"ContainerStarted","Data":"bb02096cfeb5432cf67e4991824438f1e916e732a8d7ff6b5e4f866c9c5c6e28"}
Sep 16 14:45:20.875785 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:20.875750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7" event={"ID":"8deea262-2fc4-4f53-ad0f-8174c5adfea1","Type":"ContainerStarted","Data":"beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f"}
Sep 16 14:45:20.876160 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:20.875928    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:20.877350 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:20.877324    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb" event={"ID":"15cab74b-508c-4f12-9e28-20e6adca900f","Type":"ContainerStarted","Data":"8f75986a3e7b2f6f4ff97322f6afde180377ccdddbdf62d22fd753bfdb64ec66"}
Sep 16 14:45:20.877456 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:20.877374    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:20.899443 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:20.899400    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7" podStartSLOduration=2.348283002 podStartE2EDuration="4.899385903s" totalImagesPullingTime="2.551102901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:20.89800867 +0000 UTC m=+598.052086996" watchObservedRunningTime="2026-09-16 14:45:20.899385903 +0000 UTC m=+598.053464189"
Sep 16 14:45:20.925299 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:20.925245    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb" podStartSLOduration=1.072437331 podStartE2EDuration="3.92522627s" totalImagesPullingTime="2.852788939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:20.923103044 +0000 UTC m=+598.077181330" watchObservedRunningTime="2026-09-16 14:45:20.92522627 +0000 UTC m=+598.079304556"
Sep 16 14:45:23.187609 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:23.187582    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:45:23.189334 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:23.189305    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:45:31.883685 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:31.883656    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 14:45:31.884039 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:31.883739    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:31.934986 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:31.934952    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-kdrh7"]
Sep 16 14:45:31.935219 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:31.935187    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7" podUID="8deea262-2fc4-4f53-ad0f-8174c5adfea1" containerName="manager" containerID="cri-o://beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f" gracePeriod=30
Sep 16 14:45:32.173838 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.173814    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:32.220733 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.220708    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-metrics-certs\") pod \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") "
Sep 16 14:45:32.220848 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.220749    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4gjs\" (UniqueName: \"kubernetes.io/projected/8deea262-2fc4-4f53-ad0f-8174c5adfea1-kube-api-access-t4gjs\") pod \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") "
Sep 16 14:45:32.220848 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.220797    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-webhook-cert\") pod \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\" (UID: \"8deea262-2fc4-4f53-ad0f-8174c5adfea1\") "
Sep 16 14:45:32.222824 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.222793    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8deea262-2fc4-4f53-ad0f-8174c5adfea1-kube-api-access-t4gjs" (OuterVolumeSpecName: "kube-api-access-t4gjs") pod "8deea262-2fc4-4f53-ad0f-8174c5adfea1" (UID: "8deea262-2fc4-4f53-ad0f-8174c5adfea1"). InnerVolumeSpecName "kube-api-access-t4gjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:45:32.222941 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.222828    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8deea262-2fc4-4f53-ad0f-8174c5adfea1" (UID: "8deea262-2fc4-4f53-ad0f-8174c5adfea1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:45:32.222941 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.222866    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8deea262-2fc4-4f53-ad0f-8174c5adfea1" (UID: "8deea262-2fc4-4f53-ad0f-8174c5adfea1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:45:32.321508 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.321485    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-webhook-cert\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:45:32.321508 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.321505    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8deea262-2fc4-4f53-ad0f-8174c5adfea1-metrics-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:45:32.321631 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.321515    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4gjs\" (UniqueName: \"kubernetes.io/projected/8deea262-2fc4-4f53-ad0f-8174c5adfea1-kube-api-access-t4gjs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:45:32.932035 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.932004    2558 generic.go:356] "Generic (PLEG): container finished" podID="8deea262-2fc4-4f53-ad0f-8174c5adfea1" containerID="beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f" exitCode=0
Sep 16 14:45:32.932435 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.932047    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7"
Sep 16 14:45:32.932435 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.932108    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7" event={"ID":"8deea262-2fc4-4f53-ad0f-8174c5adfea1","Type":"ContainerDied","Data":"beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f"}
Sep 16 14:45:32.932435 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.932152    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cbd6bdf6-kdrh7" event={"ID":"8deea262-2fc4-4f53-ad0f-8174c5adfea1","Type":"ContainerDied","Data":"f60a638bb540a5bcf93e3a09e0a0513530a808129e5f25101a7f90d1a5703a75"}
Sep 16 14:45:32.932435 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.932190    2558 scope.go:117] "RemoveContainer" containerID="beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f"
Sep 16 14:45:32.940666 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.940503    2558 scope.go:117] "RemoveContainer" containerID="beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f"
Sep 16 14:45:32.940750 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:45:32.940736    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f\": container with ID starting with beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f not found: ID does not exist" containerID="beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f"
Sep 16 14:45:32.940788 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.940757    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f"} err="failed to get container status \"beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f\": rpc error: code = NotFound desc = could not find container \"beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f\": container with ID starting with beb0b2cc4661e1e92ce7593354dc89c276097a2b5b3efb2f46db8593a1c4330f not found: ID does not exist"
Sep 16 14:45:32.952828 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.952805    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-kdrh7"]
Sep 16 14:45:32.956975 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:32.956956    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-kdrh7"]
Sep 16 14:45:33.298665 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:33.298596    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8deea262-2fc4-4f53-ad0f-8174c5adfea1" path="/var/lib/kubelet/pods/8deea262-2fc4-4f53-ad0f-8174c5adfea1/volumes"
Sep 16 14:45:53.588768 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.588696    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"]
Sep 16 14:45:53.591021 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.588999    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8deea262-2fc4-4f53-ad0f-8174c5adfea1" containerName="manager"
Sep 16 14:45:53.591021 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.589010    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8deea262-2fc4-4f53-ad0f-8174c5adfea1" containerName="manager"
Sep 16 14:45:53.591021 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.589065    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8deea262-2fc4-4f53-ad0f-8174c5adfea1" containerName="manager"
Sep 16 14:45:53.591787 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.591772    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"
Sep 16 14:45:53.594336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.594314    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4t4zb\""
Sep 16 14:45:53.594460 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.594442    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 14:45:53.602839 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.602808    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"]
Sep 16 14:45:53.620510 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.620485    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"]
Sep 16 14:45:53.623503 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.623485    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"
Sep 16 14:45:53.633770 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.633747    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"]
Sep 16 14:45:53.699516 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.699482    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0b07a0-1590-46ec-bc89-937aeeb72f6f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4z9lb\" (UID: \"4b0b07a0-1590-46ec-bc89-937aeeb72f6f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"
Sep 16 14:45:53.699714 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.699544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7fbf47-02a6-4b60-b02f-4ca4fb581dba-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2zpjm\" (UID: \"8f7fbf47-02a6-4b60-b02f-4ca4fb581dba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"
Sep 16 14:45:53.699806 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.699698    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsztx\" (UniqueName: \"kubernetes.io/projected/8f7fbf47-02a6-4b60-b02f-4ca4fb581dba-kube-api-access-tsztx\") pod \"payload-pre-processing-59fbc47bf8-2zpjm\" (UID: \"8f7fbf47-02a6-4b60-b02f-4ca4fb581dba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"
Sep 16 14:45:53.699806 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.699792    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpfzs\" (UniqueName: \"kubernetes.io/projected/4b0b07a0-1590-46ec-bc89-937aeeb72f6f-kube-api-access-fpfzs\") pod \"payload-processing-5b54b7fbb7-4z9lb\" (UID: \"4b0b07a0-1590-46ec-bc89-937aeeb72f6f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"
Sep 16 14:45:53.800348 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.800308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0b07a0-1590-46ec-bc89-937aeeb72f6f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4z9lb\" (UID: \"4b0b07a0-1590-46ec-bc89-937aeeb72f6f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"
Sep 16 14:45:53.800523 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.800357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7fbf47-02a6-4b60-b02f-4ca4fb581dba-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2zpjm\" (UID: \"8f7fbf47-02a6-4b60-b02f-4ca4fb581dba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"
Sep 16 14:45:53.800523 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.800386    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsztx\" (UniqueName: \"kubernetes.io/projected/8f7fbf47-02a6-4b60-b02f-4ca4fb581dba-kube-api-access-tsztx\") pod \"payload-pre-processing-59fbc47bf8-2zpjm\" (UID: \"8f7fbf47-02a6-4b60-b02f-4ca4fb581dba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"
Sep 16 14:45:53.800523 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.800418    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpfzs\" (UniqueName: \"kubernetes.io/projected/4b0b07a0-1590-46ec-bc89-937aeeb72f6f-kube-api-access-fpfzs\") pod \"payload-processing-5b54b7fbb7-4z9lb\" (UID: \"4b0b07a0-1590-46ec-bc89-937aeeb72f6f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"
Sep 16 14:45:53.801074 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.801054    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0b07a0-1590-46ec-bc89-937aeeb72f6f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-4z9lb\" (UID: \"4b0b07a0-1590-46ec-bc89-937aeeb72f6f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"
Sep 16 14:45:53.801243 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.801220    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f7fbf47-02a6-4b60-b02f-4ca4fb581dba-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2zpjm\" (UID: \"8f7fbf47-02a6-4b60-b02f-4ca4fb581dba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"
Sep 16 14:45:53.812062 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.809614    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsztx\" (UniqueName: \"kubernetes.io/projected/8f7fbf47-02a6-4b60-b02f-4ca4fb581dba-kube-api-access-tsztx\") pod \"payload-pre-processing-59fbc47bf8-2zpjm\" (UID: \"8f7fbf47-02a6-4b60-b02f-4ca4fb581dba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"
Sep 16 14:45:53.812062 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.809671    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpfzs\" (UniqueName: \"kubernetes.io/projected/4b0b07a0-1590-46ec-bc89-937aeeb72f6f-kube-api-access-fpfzs\") pod \"payload-processing-5b54b7fbb7-4z9lb\" (UID: \"4b0b07a0-1590-46ec-bc89-937aeeb72f6f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"
Sep 16 14:45:53.901601 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.901513    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"
Sep 16 14:45:53.932792 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:53.932755    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"
Sep 16 14:45:54.206022 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:54.205999    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"]
Sep 16 14:45:54.207722 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:45:54.207696    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7fbf47_02a6_4b60_b02f_4ca4fb581dba.slice/crio-393743bdda69092158f64c6ac0d4cddb00d76d7f3b232f12348db3ba0819e465 WatchSource:0}: Error finding container 393743bdda69092158f64c6ac0d4cddb00d76d7f3b232f12348db3ba0819e465: Status 404 returned error can't find the container with id 393743bdda69092158f64c6ac0d4cddb00d76d7f3b232f12348db3ba0819e465
Sep 16 14:45:54.209722 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:54.209706    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:45:54.224230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:54.224211    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"]
Sep 16 14:45:54.226583 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:45:54.226548    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0b07a0_1590_46ec_bc89_937aeeb72f6f.slice/crio-ee2566d4a9cd63ae59c40529897eadb0c779908aada240fe534d0ae62030d269 WatchSource:0}: Error finding container ee2566d4a9cd63ae59c40529897eadb0c779908aada240fe534d0ae62030d269: Status 404 returned error can't find the container with id ee2566d4a9cd63ae59c40529897eadb0c779908aada240fe534d0ae62030d269
Sep 16 14:45:55.033261 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:55.033220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb" event={"ID":"4b0b07a0-1590-46ec-bc89-937aeeb72f6f","Type":"ContainerStarted","Data":"ee2566d4a9cd63ae59c40529897eadb0c779908aada240fe534d0ae62030d269"}
Sep 16 14:45:55.034730 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:55.034700    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm" event={"ID":"8f7fbf47-02a6-4b60-b02f-4ca4fb581dba","Type":"ContainerStarted","Data":"393743bdda69092158f64c6ac0d4cddb00d76d7f3b232f12348db3ba0819e465"}
Sep 16 14:45:56.041623 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:56.041534    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb" event={"ID":"4b0b07a0-1590-46ec-bc89-937aeeb72f6f","Type":"ContainerStarted","Data":"4e818d02acd87c8fda135379a79b4a59502f354934e5ec626d774072ebace530"}
Sep 16 14:45:56.042041 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:56.041636    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"
Sep 16 14:45:56.043247 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:56.043226    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm" event={"ID":"8f7fbf47-02a6-4b60-b02f-4ca4fb581dba","Type":"ContainerStarted","Data":"2bba93ede43aeaec21a564c38994c5147024ce34c072c9896299465e555f8266"}
Sep 16 14:45:56.043320 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:56.043277    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"
Sep 16 14:45:56.060447 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:56.060405    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb" podStartSLOduration=1.697053104 podStartE2EDuration="3.060393217s" totalImagesPullingTime="1.363340113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:56.058054343 +0000 UTC m=+633.212132658" watchObservedRunningTime="2026-09-16 14:45:56.060393217 +0000 UTC m=+633.214471502"
Sep 16 14:45:56.074029 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:45:56.073982    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm" podStartSLOduration=1.703347329 podStartE2EDuration="3.073967312s" totalImagesPullingTime="1.370619983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:45:56.073143592 +0000 UTC m=+633.227221882" watchObservedRunningTime="2026-09-16 14:45:56.073967312 +0000 UTC m=+633.228045600"
Sep 16 14:46:07.049350 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:07.049315    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-4z9lb"
Sep 16 14:46:07.049706 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:07.049388    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zpjm"
Sep 16 14:46:28.418603 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.418569    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"]
Sep 16 14:46:28.425314 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.425296    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.428497 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.428453    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 14:46:28.428497 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.428467    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 14:46:28.428715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.428536    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-znlj6\""
Sep 16 14:46:28.428715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.428536    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 16 14:46:28.437950 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.437923    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"]
Sep 16 14:46:28.498108 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.498065    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.498108 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.498109    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.498329 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.498133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb9w6\" (UniqueName: \"kubernetes.io/projected/09827ed2-f19d-4840-988d-441e9530b3a1-kube-api-access-lb9w6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.498329 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.498236    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.498329 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.498295    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.498329 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.498314    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09827ed2-f19d-4840-988d-441e9530b3a1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.598809 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.598768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.598809 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.598808    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.599034 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.598836    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb9w6\" (UniqueName: \"kubernetes.io/projected/09827ed2-f19d-4840-988d-441e9530b3a1-kube-api-access-lb9w6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.599034 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.598880    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.599034 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.598905    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.599034 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.598925    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09827ed2-f19d-4840-988d-441e9530b3a1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.599387 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.599359    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.599446 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.599419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.599505 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.599463    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.601259 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.601233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09827ed2-f19d-4840-988d-441e9530b3a1-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.601416 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.601400    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09827ed2-f19d-4840-988d-441e9530b3a1-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.619427 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.619403    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb9w6\" (UniqueName: \"kubernetes.io/projected/09827ed2-f19d-4840-988d-441e9530b3a1-kube-api-access-lb9w6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9qxcq\" (UID: \"09827ed2-f19d-4840-988d-441e9530b3a1\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:28.736333 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:28.736241    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:29.008316 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:29.008279    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"]
Sep 16 14:46:29.009951 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:46:29.009928    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09827ed2_f19d_4840_988d_441e9530b3a1.slice/crio-684b551e235fd47dc523f6dc057f12885339823d000f66d14264b60b5cce60a9 WatchSource:0}: Error finding container 684b551e235fd47dc523f6dc057f12885339823d000f66d14264b60b5cce60a9: Status 404 returned error can't find the container with id 684b551e235fd47dc523f6dc057f12885339823d000f66d14264b60b5cce60a9
Sep 16 14:46:29.193550 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:29.193504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq" event={"ID":"09827ed2-f19d-4840-988d-441e9530b3a1","Type":"ContainerStarted","Data":"684b551e235fd47dc523f6dc057f12885339823d000f66d14264b60b5cce60a9"}
Sep 16 14:46:31.204424 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:31.204387    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq" event={"ID":"09827ed2-f19d-4840-988d-441e9530b3a1","Type":"ContainerStarted","Data":"258fc89da9fa00d6d31116646d91b8b2fca5aa9f4466084c563402c05d765114"}
Sep 16 14:46:31.226917 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:31.226869    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq" podStartSLOduration=2.077861119 podStartE2EDuration="3.226852588s" totalImagesPullingTime="1.148991469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:46:31.225350904 +0000 UTC m=+668.379429191" watchObservedRunningTime="2026-09-16 14:46:31.226852588 +0000 UTC m=+668.380930874"
Sep 16 14:46:38.736708 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:38.736666    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:38.737149 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:38.736816    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:38.749388 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:38.749366    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:46:39.250530 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:46:39.250501    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9qxcq"
Sep 16 14:47:23.816546 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.816477    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"]
Sep 16 14:47:23.819746 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.819728    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.823876 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.823848    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 16 14:47:23.830671 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.830647    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"]
Sep 16 14:47:23.896860 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.896831    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.896860 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.896864    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.897037 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.896883    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.897037 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.896938    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a50414-7ce6-4f89-a007-d09496a32bd3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.897037 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.896985    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.897037 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.897019    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxcj\" (UniqueName: \"kubernetes.io/projected/90a50414-7ce6-4f89-a007-d09496a32bd3-kube-api-access-wgxcj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.997673 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.997637    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.997834 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.997689    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxcj\" (UniqueName: \"kubernetes.io/projected/90a50414-7ce6-4f89-a007-d09496a32bd3-kube-api-access-wgxcj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.997834 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.997756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.997834 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.997781    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.997834 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.997806    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.998052 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.997835    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a50414-7ce6-4f89-a007-d09496a32bd3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.998052 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.998004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.998202 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.998090    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:23.998202 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:23.998191    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:24.000028 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:24.000008    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a50414-7ce6-4f89-a007-d09496a32bd3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:24.000345 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:24.000327    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a50414-7ce6-4f89-a007-d09496a32bd3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:24.006754 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:24.006732    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxcj\" (UniqueName: \"kubernetes.io/projected/90a50414-7ce6-4f89-a007-d09496a32bd3-kube-api-access-wgxcj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f48bt\" (UID: \"90a50414-7ce6-4f89-a007-d09496a32bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:24.132589 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:24.132514    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:24.401756 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:24.401709    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"]
Sep 16 14:47:24.404015 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:47:24.403984    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a50414_7ce6_4f89_a007_d09496a32bd3.slice/crio-e657e2749cf6342e950e59e3cafa7bc53b20fb048e8d741efcacfb909a421a8b WatchSource:0}: Error finding container e657e2749cf6342e950e59e3cafa7bc53b20fb048e8d741efcacfb909a421a8b: Status 404 returned error can't find the container with id e657e2749cf6342e950e59e3cafa7bc53b20fb048e8d741efcacfb909a421a8b
Sep 16 14:47:24.448973 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:24.448944    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt" event={"ID":"90a50414-7ce6-4f89-a007-d09496a32bd3","Type":"ContainerStarted","Data":"e657e2749cf6342e950e59e3cafa7bc53b20fb048e8d741efcacfb909a421a8b"}
Sep 16 14:47:25.455566 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:25.455531    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt" event={"ID":"90a50414-7ce6-4f89-a007-d09496a32bd3","Type":"ContainerStarted","Data":"773c36f4f1736312e1a43059c0b335448aec2b5df4671f7191e11e067b93370a"}
Sep 16 14:47:25.480518 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:25.480470    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt" podStartSLOduration=2.22075382 podStartE2EDuration="2.480454246s" totalImagesPullingTime="259.700426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:47:25.478874418 +0000 UTC m=+722.632952706" watchObservedRunningTime="2026-09-16 14:47:25.480454246 +0000 UTC m=+722.634532564"
Sep 16 14:47:34.133376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:34.133320    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:34.133871 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:34.133410    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:34.145973 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:34.145943    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:34.506786 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:34.506715    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f48bt"
Sep 16 14:47:38.508141 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.508108    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"]
Sep 16 14:47:38.511586 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.511563    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.514409 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.514386    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 14:47:38.524775 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.524755    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"]
Sep 16 14:47:38.628798 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.628766    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zkwb\" (UniqueName: \"kubernetes.io/projected/f0b35c03-838c-4d10-926f-64d1420946df-kube-api-access-6zkwb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.628798 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.628804    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.629009 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.628828    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.629009 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.628856    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b35c03-838c-4d10-926f-64d1420946df-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.629009 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.628913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.629009 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.628957    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.729741 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.729707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b35c03-838c-4d10-926f-64d1420946df-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.729741 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.729744    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.729992 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.729771    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.729992 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.729809    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zkwb\" (UniqueName: \"kubernetes.io/projected/f0b35c03-838c-4d10-926f-64d1420946df-kube-api-access-6zkwb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.729992 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.729829    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.729992 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.729851    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.730238 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.730112    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.730302 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.730264    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.730384 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.730345    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.732157 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.732137    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b35c03-838c-4d10-926f-64d1420946df-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.732234 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.732221    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b35c03-838c-4d10-926f-64d1420946df-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.738234 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.738214    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zkwb\" (UniqueName: \"kubernetes.io/projected/f0b35c03-838c-4d10-926f-64d1420946df-kube-api-access-6zkwb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tht5l\" (UID: \"f0b35c03-838c-4d10-926f-64d1420946df\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:38.822300 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:38.822262    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:39.096706 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:39.096680    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"]
Sep 16 14:47:39.099387 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:47:39.099354    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0b35c03_838c_4d10_926f_64d1420946df.slice/crio-875786f940c0e5421eb7fc681fd35ccf06c86105ecb3d61b52429e5fd859a81a WatchSource:0}: Error finding container 875786f940c0e5421eb7fc681fd35ccf06c86105ecb3d61b52429e5fd859a81a: Status 404 returned error can't find the container with id 875786f940c0e5421eb7fc681fd35ccf06c86105ecb3d61b52429e5fd859a81a
Sep 16 14:47:39.521253 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:39.521153    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l" event={"ID":"f0b35c03-838c-4d10-926f-64d1420946df","Type":"ContainerStarted","Data":"91060548344b695b39b9510e06f5aaf93129be27991ca9624b8d63883323d0d5"}
Sep 16 14:47:39.521253 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:39.521213    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l" event={"ID":"f0b35c03-838c-4d10-926f-64d1420946df","Type":"ContainerStarted","Data":"875786f940c0e5421eb7fc681fd35ccf06c86105ecb3d61b52429e5fd859a81a"}
Sep 16 14:47:39.542136 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:39.542089    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l" podStartSLOduration=1.359724655 podStartE2EDuration="1.542072502s" totalImagesPullingTime="182.347847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:47:39.540901312 +0000 UTC m=+736.694979611" watchObservedRunningTime="2026-09-16 14:47:39.542072502 +0000 UTC m=+736.696150789"
Sep 16 14:47:48.822892 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:48.822857    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:48.823453 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:48.822912    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:48.835294 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:48.835272    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:47:49.579488 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:47:49.579458    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tht5l"
Sep 16 14:49:04.681498 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:04.681462    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6998bc65dd-7n6g4"]
Sep 16 14:49:04.684730 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:04.684701    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6998bc65dd-7n6g4"
Sep 16 14:49:04.715558 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:04.715535    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6998bc65dd-7n6g4"]
Sep 16 14:49:04.788702 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:04.788666    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9adff22f-71ea-4168-b8af-c2a3d22ab2d9-tls-cert\") pod \"authorino-6998bc65dd-7n6g4\" (UID: \"9adff22f-71ea-4168-b8af-c2a3d22ab2d9\") " pod="kuadrant-system/authorino-6998bc65dd-7n6g4"
Sep 16 14:49:04.788871 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:04.788766    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzbx2\" (UniqueName: \"kubernetes.io/projected/9adff22f-71ea-4168-b8af-c2a3d22ab2d9-kube-api-access-qzbx2\") pod \"authorino-6998bc65dd-7n6g4\" (UID: \"9adff22f-71ea-4168-b8af-c2a3d22ab2d9\") " pod="kuadrant-system/authorino-6998bc65dd-7n6g4"
Sep 16 14:49:04.889244 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:04.889210    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzbx2\" (UniqueName: \"kubernetes.io/projected/9adff22f-71ea-4168-b8af-c2a3d22ab2d9-kube-api-access-qzbx2\") pod \"authorino-6998bc65dd-7n6g4\" (UID: \"9adff22f-71ea-4168-b8af-c2a3d22ab2d9\") " pod="kuadrant-system/authorino-6998bc65dd-7n6g4"
Sep 16 14:49:04.889402 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:04.889262    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9adff22f-71ea-4168-b8af-c2a3d22ab2d9-tls-cert\") pod \"authorino-6998bc65dd-7n6g4\" (UID: \"9adff22f-71ea-4168-b8af-c2a3d22ab2d9\") " pod="kuadrant-system/authorino-6998bc65dd-7n6g4"
Sep 16 14:49:04.891594 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:04.891577    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9adff22f-71ea-4168-b8af-c2a3d22ab2d9-tls-cert\") pod \"authorino-6998bc65dd-7n6g4\" (UID: \"9adff22f-71ea-4168-b8af-c2a3d22ab2d9\") " pod="kuadrant-system/authorino-6998bc65dd-7n6g4"
Sep 16 14:49:04.897413 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:04.897389    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzbx2\" (UniqueName: \"kubernetes.io/projected/9adff22f-71ea-4168-b8af-c2a3d22ab2d9-kube-api-access-qzbx2\") pod \"authorino-6998bc65dd-7n6g4\" (UID: \"9adff22f-71ea-4168-b8af-c2a3d22ab2d9\") " pod="kuadrant-system/authorino-6998bc65dd-7n6g4"
Sep 16 14:49:04.994062 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:04.993989    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6998bc65dd-7n6g4"
Sep 16 14:49:05.263803 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:05.263668    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6998bc65dd-7n6g4"]
Sep 16 14:49:05.266454 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:49:05.266427    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9adff22f_71ea_4168_b8af_c2a3d22ab2d9.slice/crio-5809f798b94924da76da94c952a18e746b6382ff7fc1d8fb4f5b56d5143d9afd WatchSource:0}: Error finding container 5809f798b94924da76da94c952a18e746b6382ff7fc1d8fb4f5b56d5143d9afd: Status 404 returned error can't find the container with id 5809f798b94924da76da94c952a18e746b6382ff7fc1d8fb4f5b56d5143d9afd
Sep 16 14:49:05.913231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:05.913193    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6998bc65dd-7n6g4" event={"ID":"9adff22f-71ea-4168-b8af-c2a3d22ab2d9","Type":"ContainerStarted","Data":"37daf71e689665057c8ab3a37c593b9ae48f9a0fdf75bc349f9e776c4004ff7f"}
Sep 16 14:49:05.913688 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:05.913239    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6998bc65dd-7n6g4" event={"ID":"9adff22f-71ea-4168-b8af-c2a3d22ab2d9","Type":"ContainerStarted","Data":"5809f798b94924da76da94c952a18e746b6382ff7fc1d8fb4f5b56d5143d9afd"}
Sep 16 14:49:05.929659 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:05.929612    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6998bc65dd-7n6g4" podStartSLOduration=1.43465936 podStartE2EDuration="1.929594059s" totalImagesPullingTime="494.934699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:05.928352046 +0000 UTC m=+823.082430332" watchObservedRunningTime="2026-09-16 14:49:05.929594059 +0000 UTC m=+823.083672345"
Sep 16 14:49:05.954202 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:05.954157    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f9cd9c497-xxzxq"]
Sep 16 14:49:05.954588 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:05.954372    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7f9cd9c497-xxzxq" podUID="6552d805-211a-46ab-8f7f-0b5dbc1b5cec" containerName="authorino" containerID="cri-o://6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893" gracePeriod=30
Sep 16 14:49:06.218970 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.218943    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f9cd9c497-xxzxq"
Sep 16 14:49:06.302113 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.302079    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n4f8\" (UniqueName: \"kubernetes.io/projected/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-kube-api-access-6n4f8\") pod \"6552d805-211a-46ab-8f7f-0b5dbc1b5cec\" (UID: \"6552d805-211a-46ab-8f7f-0b5dbc1b5cec\") "
Sep 16 14:49:06.302322 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.302203    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-tls-cert\") pod \"6552d805-211a-46ab-8f7f-0b5dbc1b5cec\" (UID: \"6552d805-211a-46ab-8f7f-0b5dbc1b5cec\") "
Sep 16 14:49:06.304556 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.304522    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-kube-api-access-6n4f8" (OuterVolumeSpecName: "kube-api-access-6n4f8") pod "6552d805-211a-46ab-8f7f-0b5dbc1b5cec" (UID: "6552d805-211a-46ab-8f7f-0b5dbc1b5cec"). InnerVolumeSpecName "kube-api-access-6n4f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:49:06.312400 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.312374    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6552d805-211a-46ab-8f7f-0b5dbc1b5cec" (UID: "6552d805-211a-46ab-8f7f-0b5dbc1b5cec"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:49:06.403571 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.403479    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n4f8\" (UniqueName: \"kubernetes.io/projected/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-kube-api-access-6n4f8\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:49:06.403571 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.403514    2558 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6552d805-211a-46ab-8f7f-0b5dbc1b5cec-tls-cert\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:49:06.926290 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.926152    2558 generic.go:356] "Generic (PLEG): container finished" podID="6552d805-211a-46ab-8f7f-0b5dbc1b5cec" containerID="6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893" exitCode=0
Sep 16 14:49:06.926290 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.926241    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f9cd9c497-xxzxq"
Sep 16 14:49:06.926830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.926291    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f9cd9c497-xxzxq" event={"ID":"6552d805-211a-46ab-8f7f-0b5dbc1b5cec","Type":"ContainerDied","Data":"6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893"}
Sep 16 14:49:06.926830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.926328    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f9cd9c497-xxzxq" event={"ID":"6552d805-211a-46ab-8f7f-0b5dbc1b5cec","Type":"ContainerDied","Data":"aa69c8dfaa16ffa5e14cc4a41b2b291ee209b7fbd1b64b320dfd91decafba08b"}
Sep 16 14:49:06.926830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.926344    2558 scope.go:117] "RemoveContainer" containerID="6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893"
Sep 16 14:49:06.954681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.954662    2558 scope.go:117] "RemoveContainer" containerID="6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893"
Sep 16 14:49:06.955307 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:49:06.955283    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893\": container with ID starting with 6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893 not found: ID does not exist" containerID="6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893"
Sep 16 14:49:06.955456 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.955433    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893"} err="failed to get container status \"6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893\": rpc error: code = NotFound desc = could not find container \"6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893\": container with ID starting with 6b61c07fab23272fb36fa6bd539ab412d3d2d97a822c37eadf1e93e1d912e893 not found: ID does not exist"
Sep 16 14:49:06.960330 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.960310    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f9cd9c497-xxzxq"]
Sep 16 14:49:06.962613 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:06.962588    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f9cd9c497-xxzxq"]
Sep 16 14:49:07.295937 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:07.295861    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6552d805-211a-46ab-8f7f-0b5dbc1b5cec" path="/var/lib/kubelet/pods/6552d805-211a-46ab-8f7f-0b5dbc1b5cec/volumes"
Sep 16 14:49:32.672898 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.672865    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"]
Sep 16 14:49:32.673332 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.673227    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6552d805-211a-46ab-8f7f-0b5dbc1b5cec" containerName="authorino"
Sep 16 14:49:32.673332 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.673239    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6552d805-211a-46ab-8f7f-0b5dbc1b5cec" containerName="authorino"
Sep 16 14:49:32.673332 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.673306    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6552d805-211a-46ab-8f7f-0b5dbc1b5cec" containerName="authorino"
Sep 16 14:49:32.677629 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.677611    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 14:49:32.680638 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.680617    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-780e10\""
Sep 16 14:49:32.688212 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.688191    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"]
Sep 16 14:49:32.702574 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.702546    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"]
Sep 16 14:49:32.705888 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.705873    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 14:49:32.708406 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.708388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-780e10-dockercfg-mw25n\""
Sep 16 14:49:32.716960 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.716940    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"]
Sep 16 14:49:32.731689 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.731666    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhgrm\" (UniqueName: \"kubernetes.io/projected/7456814a-59b2-42ca-a415-9ffd7b975e91-kube-api-access-bhgrm\") pod \"payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr\" (UID: \"7456814a-59b2-42ca-a415-9ffd7b975e91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 14:49:32.731800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.731708    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7456814a-59b2-42ca-a415-9ffd7b975e91-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr\" (UID: \"7456814a-59b2-42ca-a415-9ffd7b975e91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 14:49:32.731800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.731736    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ztdf\" (UniqueName: \"kubernetes.io/projected/07cfd061-a487-4331-9114-dfa8e3cfc1a1-kube-api-access-5ztdf\") pod \"payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d\" (UID: \"07cfd061-a487-4331-9114-dfa8e3cfc1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 14:49:32.731889 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.731806    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07cfd061-a487-4331-9114-dfa8e3cfc1a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d\" (UID: \"07cfd061-a487-4331-9114-dfa8e3cfc1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 14:49:32.832766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.832724    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07cfd061-a487-4331-9114-dfa8e3cfc1a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d\" (UID: \"07cfd061-a487-4331-9114-dfa8e3cfc1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 14:49:32.833041 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.832802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhgrm\" (UniqueName: \"kubernetes.io/projected/7456814a-59b2-42ca-a415-9ffd7b975e91-kube-api-access-bhgrm\") pod \"payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr\" (UID: \"7456814a-59b2-42ca-a415-9ffd7b975e91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 14:49:32.833041 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.832837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7456814a-59b2-42ca-a415-9ffd7b975e91-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr\" (UID: \"7456814a-59b2-42ca-a415-9ffd7b975e91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 14:49:32.833041 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.832868    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ztdf\" (UniqueName: \"kubernetes.io/projected/07cfd061-a487-4331-9114-dfa8e3cfc1a1-kube-api-access-5ztdf\") pod \"payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d\" (UID: \"07cfd061-a487-4331-9114-dfa8e3cfc1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 14:49:32.833468 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.833448    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07cfd061-a487-4331-9114-dfa8e3cfc1a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d\" (UID: \"07cfd061-a487-4331-9114-dfa8e3cfc1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 14:49:32.833516 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.833502    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7456814a-59b2-42ca-a415-9ffd7b975e91-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr\" (UID: \"7456814a-59b2-42ca-a415-9ffd7b975e91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 14:49:32.847797 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.847767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ztdf\" (UniqueName: \"kubernetes.io/projected/07cfd061-a487-4331-9114-dfa8e3cfc1a1-kube-api-access-5ztdf\") pod \"payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d\" (UID: \"07cfd061-a487-4331-9114-dfa8e3cfc1a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 14:49:32.847900 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.847883    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhgrm\" (UniqueName: \"kubernetes.io/projected/7456814a-59b2-42ca-a415-9ffd7b975e91-kube-api-access-bhgrm\") pod \"payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr\" (UID: \"7456814a-59b2-42ca-a415-9ffd7b975e91\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 14:49:32.989025 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:32.988944    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 14:49:33.015314 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:33.015282    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 14:49:33.262245 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:33.262208    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"]
Sep 16 14:49:33.264470 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:49:33.264442    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07cfd061_a487_4331_9114_dfa8e3cfc1a1.slice/crio-3a954a853e67d39412dafe161e844aa2c107867a658245c424c864b9019052e8 WatchSource:0}: Error finding container 3a954a853e67d39412dafe161e844aa2c107867a658245c424c864b9019052e8: Status 404 returned error can't find the container with id 3a954a853e67d39412dafe161e844aa2c107867a658245c424c864b9019052e8
Sep 16 14:49:33.305677 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:33.305659    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"]
Sep 16 14:49:33.307484 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:49:33.307452    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7456814a_59b2_42ca_a415_9ffd7b975e91.slice/crio-e70fb80686b4e49ab8b4bd177cf1e07ddd3947a356ddffe4a784e2a53eed347f WatchSource:0}: Error finding container e70fb80686b4e49ab8b4bd177cf1e07ddd3947a356ddffe4a784e2a53eed347f: Status 404 returned error can't find the container with id e70fb80686b4e49ab8b4bd177cf1e07ddd3947a356ddffe4a784e2a53eed347f
Sep 16 14:49:34.059300 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.059141    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr" event={"ID":"7456814a-59b2-42ca-a415-9ffd7b975e91","Type":"ContainerStarted","Data":"c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a"}
Sep 16 14:49:34.059300 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.059296    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 14:49:34.059803 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.059334    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr" event={"ID":"7456814a-59b2-42ca-a415-9ffd7b975e91","Type":"ContainerStarted","Data":"e70fb80686b4e49ab8b4bd177cf1e07ddd3947a356ddffe4a784e2a53eed347f"}
Sep 16 14:49:34.060846 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.060819    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d" event={"ID":"07cfd061-a487-4331-9114-dfa8e3cfc1a1","Type":"ContainerStarted","Data":"a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812"}
Sep 16 14:49:34.060975 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.060869    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d" event={"ID":"07cfd061-a487-4331-9114-dfa8e3cfc1a1","Type":"ContainerStarted","Data":"3a954a853e67d39412dafe161e844aa2c107867a658245c424c864b9019052e8"}
Sep 16 14:49:34.060975 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.060905    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 14:49:34.082350 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.082301    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr" podStartSLOduration=2.08228023 podStartE2EDuration="2.08228023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:34.081350351 +0000 UTC m=+851.235428646" watchObservedRunningTime="2026-09-16 14:49:34.08228023 +0000 UTC m=+851.236358519"
Sep 16 14:49:34.099003 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.098960    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d" podStartSLOduration=2.098945717 podStartE2EDuration="2.098945717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:34.097878572 +0000 UTC m=+851.251956859" watchObservedRunningTime="2026-09-16 14:49:34.098945717 +0000 UTC m=+851.253024003"
Sep 16 14:49:34.463055 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.462979    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"]
Sep 16 14:49:34.466077 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.466060    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:49:34.468651 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.468629    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-78e52a\""
Sep 16 14:49:34.468741 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.468685    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-78e52a-dockercfg-8rr97\""
Sep 16 14:49:34.478543 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.478522    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"]
Sep 16 14:49:34.494923 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.494901    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"]
Sep 16 14:49:34.498247 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.498229    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:49:34.509327 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.509307    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"]
Sep 16 14:49:34.548641 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.548612    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddx2\" (UniqueName: \"kubernetes.io/projected/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-kube-api-access-lddx2\") pod \"payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst\" (UID: \"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:49:34.548763 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.548641    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78951ca2-2329-4c12-b181-5f4543fe4fef-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67\" (UID: \"78951ca2-2329-4c12-b181-5f4543fe4fef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:49:34.548763 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.548710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmp6n\" (UniqueName: \"kubernetes.io/projected/78951ca2-2329-4c12-b181-5f4543fe4fef-kube-api-access-xmp6n\") pod \"payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67\" (UID: \"78951ca2-2329-4c12-b181-5f4543fe4fef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:49:34.548848 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.548768    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst\" (UID: \"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:49:34.649716 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.649679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddx2\" (UniqueName: \"kubernetes.io/projected/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-kube-api-access-lddx2\") pod \"payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst\" (UID: \"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:49:34.649716 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.649721    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78951ca2-2329-4c12-b181-5f4543fe4fef-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67\" (UID: \"78951ca2-2329-4c12-b181-5f4543fe4fef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:49:34.649921 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.649763    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmp6n\" (UniqueName: \"kubernetes.io/projected/78951ca2-2329-4c12-b181-5f4543fe4fef-kube-api-access-xmp6n\") pod \"payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67\" (UID: \"78951ca2-2329-4c12-b181-5f4543fe4fef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:49:34.649921 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.649802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst\" (UID: \"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:49:34.650544 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.650514    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78951ca2-2329-4c12-b181-5f4543fe4fef-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67\" (UID: \"78951ca2-2329-4c12-b181-5f4543fe4fef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:49:34.650664 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.650561    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst\" (UID: \"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:49:34.660471 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.660435    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddx2\" (UniqueName: \"kubernetes.io/projected/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-kube-api-access-lddx2\") pod \"payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst\" (UID: \"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:49:34.660471 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.660461    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmp6n\" (UniqueName: \"kubernetes.io/projected/78951ca2-2329-4c12-b181-5f4543fe4fef-kube-api-access-xmp6n\") pod \"payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67\" (UID: \"78951ca2-2329-4c12-b181-5f4543fe4fef\") " pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:49:34.776831 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.776742    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:49:34.807991 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:34.807960    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:49:35.051909 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.051879    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"]
Sep 16 14:49:35.053556 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:49:35.053525    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25842f7c_d1d7_4c83_9dbe_c3da0a4cffea.slice/crio-ed7e11234fae67f612679575f0532989c6356e939588180f8357ce0669ca8fc6 WatchSource:0}: Error finding container ed7e11234fae67f612679575f0532989c6356e939588180f8357ce0669ca8fc6: Status 404 returned error can't find the container with id ed7e11234fae67f612679575f0532989c6356e939588180f8357ce0669ca8fc6
Sep 16 14:49:35.065994 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.065964    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst" event={"ID":"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea","Type":"ContainerStarted","Data":"ed7e11234fae67f612679575f0532989c6356e939588180f8357ce0669ca8fc6"}
Sep 16 14:49:35.096648 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.096627    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"]
Sep 16 14:49:35.098337 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:49:35.098277    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78951ca2_2329_4c12_b181_5f4543fe4fef.slice/crio-f494c7aa9245f31986132d9f3d049414ab9933b1f3751bab9765a6be0d00e46f WatchSource:0}: Error finding container f494c7aa9245f31986132d9f3d049414ab9933b1f3751bab9765a6be0d00e46f: Status 404 returned error can't find the container with id f494c7aa9245f31986132d9f3d049414ab9933b1f3751bab9765a6be0d00e46f
Sep 16 14:49:35.730352 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.730321    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"]
Sep 16 14:49:35.733847 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.733830    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 14:49:35.736607 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.736589    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e33b3f\""
Sep 16 14:49:35.736847 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.736831    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-e33b3f-dockercfg-djpwp\""
Sep 16 14:49:35.747443 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.747419    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"]
Sep 16 14:49:35.862995 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.862955    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562dfebc-3b43-488c-affe-0854a5f9faae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc\" (UID: \"562dfebc-3b43-488c-affe-0854a5f9faae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 14:49:35.862995 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.862993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5r8b\" (UniqueName: \"kubernetes.io/projected/562dfebc-3b43-488c-affe-0854a5f9faae-kube-api-access-x5r8b\") pod \"payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc\" (UID: \"562dfebc-3b43-488c-affe-0854a5f9faae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 14:49:35.964159 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.964121    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562dfebc-3b43-488c-affe-0854a5f9faae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc\" (UID: \"562dfebc-3b43-488c-affe-0854a5f9faae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 14:49:35.964350 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.964186    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5r8b\" (UniqueName: \"kubernetes.io/projected/562dfebc-3b43-488c-affe-0854a5f9faae-kube-api-access-x5r8b\") pod \"payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc\" (UID: \"562dfebc-3b43-488c-affe-0854a5f9faae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 14:49:35.964857 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.964832    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562dfebc-3b43-488c-affe-0854a5f9faae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc\" (UID: \"562dfebc-3b43-488c-affe-0854a5f9faae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 14:49:35.980336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:35.980311    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5r8b\" (UniqueName: \"kubernetes.io/projected/562dfebc-3b43-488c-affe-0854a5f9faae-kube-api-access-x5r8b\") pod \"payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc\" (UID: \"562dfebc-3b43-488c-affe-0854a5f9faae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 14:49:36.045187 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:36.045097    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 14:49:36.073147 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:36.073114    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst" event={"ID":"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea","Type":"ContainerStarted","Data":"c8ab409c9bc9c607b5bd01ef3371b2758f8c70fb6e8bde4174890a1d67d730eb"}
Sep 16 14:49:36.073579 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:36.073218    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:49:36.074857 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:36.074836    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67" event={"ID":"78951ca2-2329-4c12-b181-5f4543fe4fef","Type":"ContainerStarted","Data":"5c22c0c5826be72d0387a7c6f03c99f4f40881657443748e1847eb3463e0fdb1"}
Sep 16 14:49:36.074996 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:36.074877    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67" event={"ID":"78951ca2-2329-4c12-b181-5f4543fe4fef","Type":"ContainerStarted","Data":"f494c7aa9245f31986132d9f3d049414ab9933b1f3751bab9765a6be0d00e46f"}
Sep 16 14:49:36.074996 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:36.074897    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:49:36.136642 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:36.136593    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67" podStartSLOduration=2.136573934 podStartE2EDuration="2.136573934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:36.13589828 +0000 UTC m=+853.289976593" watchObservedRunningTime="2026-09-16 14:49:36.136573934 +0000 UTC m=+853.290652224"
Sep 16 14:49:36.137596 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:36.137559    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst" podStartSLOduration=2.137548625 podStartE2EDuration="2.137548625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:36.11111061 +0000 UTC m=+853.265188908" watchObservedRunningTime="2026-09-16 14:49:36.137548625 +0000 UTC m=+853.291626909"
Sep 16 14:49:36.325986 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:36.325959    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"]
Sep 16 14:49:36.328008 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:49:36.327981    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562dfebc_3b43_488c_affe_0854a5f9faae.slice/crio-2c8476333717c79a1b2b79c4ec190981e9dc483b443f120e5751f3f9b5756d58 WatchSource:0}: Error finding container 2c8476333717c79a1b2b79c4ec190981e9dc483b443f120e5751f3f9b5756d58: Status 404 returned error can't find the container with id 2c8476333717c79a1b2b79c4ec190981e9dc483b443f120e5751f3f9b5756d58
Sep 16 14:49:37.083245 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:37.083186    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc" event={"ID":"562dfebc-3b43-488c-affe-0854a5f9faae","Type":"ContainerStarted","Data":"1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a"}
Sep 16 14:49:37.083245 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:37.083259    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc" event={"ID":"562dfebc-3b43-488c-affe-0854a5f9faae","Type":"ContainerStarted","Data":"2c8476333717c79a1b2b79c4ec190981e9dc483b443f120e5751f3f9b5756d58"}
Sep 16 14:49:37.083785 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:37.083293    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 14:49:37.103005 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:37.102946    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc" podStartSLOduration=2.102927064 podStartE2EDuration="2.102927064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:37.099718341 +0000 UTC m=+854.253796629" watchObservedRunningTime="2026-09-16 14:49:37.102927064 +0000 UTC m=+854.257005354"
Sep 16 14:49:43.303606 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.303572    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"]
Sep 16 14:49:43.307361 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.307344    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:49:43.310199 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.310157    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0252b3a9\""
Sep 16 14:49:43.310335 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.310318    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0252b3a9-dockercfg-fxktf\""
Sep 16 14:49:43.319118 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.319097    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"]
Sep 16 14:49:43.349066 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.349035    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"]
Sep 16 14:49:43.352736 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.352710    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:49:43.363506 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.363482    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"]
Sep 16 14:49:43.434395 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.434362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697fd466-de3f-4e3b-85f1-85d9deeff0c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f\" (UID: \"697fd466-de3f-4e3b-85f1-85d9deeff0c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:49:43.434532 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.434422    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qx9t\" (UniqueName: \"kubernetes.io/projected/697fd466-de3f-4e3b-85f1-85d9deeff0c8-kube-api-access-7qx9t\") pod \"payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f\" (UID: \"697fd466-de3f-4e3b-85f1-85d9deeff0c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:49:43.434583 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.434530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5jr\" (UniqueName: \"kubernetes.io/projected/0e4547e3-0496-4514-a490-d6688b1a412f-kube-api-access-2q5jr\") pod \"payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82\" (UID: \"0e4547e3-0496-4514-a490-d6688b1a412f\") " pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:49:43.434626 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.434595    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4547e3-0496-4514-a490-d6688b1a412f-plugins-config-volume\") pod \"payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82\" (UID: \"0e4547e3-0496-4514-a490-d6688b1a412f\") " pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:49:43.535435 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.535392    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697fd466-de3f-4e3b-85f1-85d9deeff0c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f\" (UID: \"697fd466-de3f-4e3b-85f1-85d9deeff0c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:49:43.535613 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.535485    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qx9t\" (UniqueName: \"kubernetes.io/projected/697fd466-de3f-4e3b-85f1-85d9deeff0c8-kube-api-access-7qx9t\") pod \"payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f\" (UID: \"697fd466-de3f-4e3b-85f1-85d9deeff0c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:49:43.535613 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.535539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5jr\" (UniqueName: \"kubernetes.io/projected/0e4547e3-0496-4514-a490-d6688b1a412f-kube-api-access-2q5jr\") pod \"payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82\" (UID: \"0e4547e3-0496-4514-a490-d6688b1a412f\") " pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:49:43.535613 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.535592    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4547e3-0496-4514-a490-d6688b1a412f-plugins-config-volume\") pod \"payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82\" (UID: \"0e4547e3-0496-4514-a490-d6688b1a412f\") " pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:49:43.536437 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.536402    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697fd466-de3f-4e3b-85f1-85d9deeff0c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f\" (UID: \"697fd466-de3f-4e3b-85f1-85d9deeff0c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:49:43.537065 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.537040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4547e3-0496-4514-a490-d6688b1a412f-plugins-config-volume\") pod \"payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82\" (UID: \"0e4547e3-0496-4514-a490-d6688b1a412f\") " pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:49:43.557945 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.557857    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5jr\" (UniqueName: \"kubernetes.io/projected/0e4547e3-0496-4514-a490-d6688b1a412f-kube-api-access-2q5jr\") pod \"payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82\" (UID: \"0e4547e3-0496-4514-a490-d6688b1a412f\") " pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:49:43.558722 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.558687    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qx9t\" (UniqueName: \"kubernetes.io/projected/697fd466-de3f-4e3b-85f1-85d9deeff0c8-kube-api-access-7qx9t\") pod \"payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f\" (UID: \"697fd466-de3f-4e3b-85f1-85d9deeff0c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:49:43.617815 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.617781    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:49:43.664333 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.664297    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:49:43.895941 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.895816    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"]
Sep 16 14:49:43.898013 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:49:43.897988    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e4547e3_0496_4514_a490_d6688b1a412f.slice/crio-b739ddf453b010d874a918a96571948b63fa0030e154703a8842239d899f87f1 WatchSource:0}: Error finding container b739ddf453b010d874a918a96571948b63fa0030e154703a8842239d899f87f1: Status 404 returned error can't find the container with id b739ddf453b010d874a918a96571948b63fa0030e154703a8842239d899f87f1
Sep 16 14:49:43.960238 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:43.960110    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"]
Sep 16 14:49:43.962334 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:49:43.962309    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697fd466_de3f_4e3b_85f1_85d9deeff0c8.slice/crio-4af1bb594e4156d0825551a76bbaca4e1f863ea99a76c39bd5378e911e1ba33e WatchSource:0}: Error finding container 4af1bb594e4156d0825551a76bbaca4e1f863ea99a76c39bd5378e911e1ba33e: Status 404 returned error can't find the container with id 4af1bb594e4156d0825551a76bbaca4e1f863ea99a76c39bd5378e911e1ba33e
Sep 16 14:49:44.128937 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:44.128893    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f" event={"ID":"697fd466-de3f-4e3b-85f1-85d9deeff0c8","Type":"ContainerStarted","Data":"590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7"}
Sep 16 14:49:44.129123 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:44.128959    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:49:44.129123 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:44.128981    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f" event={"ID":"697fd466-de3f-4e3b-85f1-85d9deeff0c8","Type":"ContainerStarted","Data":"4af1bb594e4156d0825551a76bbaca4e1f863ea99a76c39bd5378e911e1ba33e"}
Sep 16 14:49:44.130524 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:44.130487    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82" event={"ID":"0e4547e3-0496-4514-a490-d6688b1a412f","Type":"ContainerStarted","Data":"e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d"}
Sep 16 14:49:44.130634 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:44.130542    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:49:44.130634 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:44.130568    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82" event={"ID":"0e4547e3-0496-4514-a490-d6688b1a412f","Type":"ContainerStarted","Data":"b739ddf453b010d874a918a96571948b63fa0030e154703a8842239d899f87f1"}
Sep 16 14:49:44.146032 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:44.145948    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f" podStartSLOduration=1.145935044 podStartE2EDuration="1.145935044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:44.144980739 +0000 UTC m=+861.299059029" watchObservedRunningTime="2026-09-16 14:49:44.145935044 +0000 UTC m=+861.300013330"
Sep 16 14:49:44.162018 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:44.161975    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82" podStartSLOduration=1.16196069 podStartE2EDuration="1.16196069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:44.160383498 +0000 UTC m=+861.314461788" watchObservedRunningTime="2026-09-16 14:49:44.16196069 +0000 UTC m=+861.316038977"
Sep 16 14:49:45.068410 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.068381    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 14:49:45.068829 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.068627    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 14:49:45.254677 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.254645    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"]
Sep 16 14:49:45.258470 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.258454    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:49:45.262697 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.262673    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-bde531\""
Sep 16 14:49:45.262888 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.262872    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-bde531-dockercfg-rhmdn\""
Sep 16 14:49:45.269853 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.269829    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"]
Sep 16 14:49:45.275391 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.275281    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"]
Sep 16 14:49:45.280494 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.280470    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:49:45.298210 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.298182    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"]
Sep 16 14:49:45.353969 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.353879    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-plugins-config-volume\") pod \"payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k\" (UID: \"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb\") " pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:49:45.353969 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.353920    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgfjc\" (UniqueName: \"kubernetes.io/projected/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-kube-api-access-dgfjc\") pod \"payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt\" (UID: \"4cb2ec3d-b596-4eb7-8e63-66d45773bae7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:49:45.354228 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.354116    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7zb\" (UniqueName: \"kubernetes.io/projected/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-kube-api-access-gv7zb\") pod \"payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k\" (UID: \"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb\") " pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:49:45.354309 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.354269    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt\" (UID: \"4cb2ec3d-b596-4eb7-8e63-66d45773bae7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:49:45.455650 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.455608    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7zb\" (UniqueName: \"kubernetes.io/projected/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-kube-api-access-gv7zb\") pod \"payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k\" (UID: \"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb\") " pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:49:45.455830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.455669    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt\" (UID: \"4cb2ec3d-b596-4eb7-8e63-66d45773bae7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:49:45.455830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.455701    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-plugins-config-volume\") pod \"payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k\" (UID: \"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb\") " pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:49:45.455830 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.455733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgfjc\" (UniqueName: \"kubernetes.io/projected/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-kube-api-access-dgfjc\") pod \"payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt\" (UID: \"4cb2ec3d-b596-4eb7-8e63-66d45773bae7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:49:45.456518 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.456492    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt\" (UID: \"4cb2ec3d-b596-4eb7-8e63-66d45773bae7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:49:45.456628 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.456496    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-plugins-config-volume\") pod \"payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k\" (UID: \"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb\") " pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:49:45.464616 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.464587    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7zb\" (UniqueName: \"kubernetes.io/projected/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-kube-api-access-gv7zb\") pod \"payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k\" (UID: \"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb\") " pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:49:45.464981 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.464959    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgfjc\" (UniqueName: \"kubernetes.io/projected/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-kube-api-access-dgfjc\") pod \"payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt\" (UID: \"4cb2ec3d-b596-4eb7-8e63-66d45773bae7\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:49:45.572773 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.572736    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:49:45.592108 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.592085    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:49:45.854431 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.854402    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"]
Sep 16 14:49:45.856117 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:49:45.856020    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb2ec3d_b596_4eb7_8e63_66d45773bae7.slice/crio-053f6561203c19f7aa3d8412910dcab86c9892365eaaa11be8388d0847767300 WatchSource:0}: Error finding container 053f6561203c19f7aa3d8412910dcab86c9892365eaaa11be8388d0847767300: Status 404 returned error can't find the container with id 053f6561203c19f7aa3d8412910dcab86c9892365eaaa11be8388d0847767300
Sep 16 14:49:45.885836 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:45.885813    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"]
Sep 16 14:49:45.895246 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:49:45.895223    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf7cd73_6e18_40da_9fa2_b40e0da7c8bb.slice/crio-0aefeeceed150f4bef9bd02e6fa899ad952adf7fce1a7355231fdc15ba3e2bc1 WatchSource:0}: Error finding container 0aefeeceed150f4bef9bd02e6fa899ad952adf7fce1a7355231fdc15ba3e2bc1: Status 404 returned error can't find the container with id 0aefeeceed150f4bef9bd02e6fa899ad952adf7fce1a7355231fdc15ba3e2bc1
Sep 16 14:49:46.143419 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:46.143324    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt" event={"ID":"4cb2ec3d-b596-4eb7-8e63-66d45773bae7","Type":"ContainerStarted","Data":"161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd"}
Sep 16 14:49:46.143419 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:46.143388    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt" event={"ID":"4cb2ec3d-b596-4eb7-8e63-66d45773bae7","Type":"ContainerStarted","Data":"053f6561203c19f7aa3d8412910dcab86c9892365eaaa11be8388d0847767300"}
Sep 16 14:49:46.143419 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:46.143419    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:49:46.144982 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:46.144950    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k" event={"ID":"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb","Type":"ContainerStarted","Data":"f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c"}
Sep 16 14:49:46.145102 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:46.144999    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k" event={"ID":"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb","Type":"ContainerStarted","Data":"0aefeeceed150f4bef9bd02e6fa899ad952adf7fce1a7355231fdc15ba3e2bc1"}
Sep 16 14:49:46.145102 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:46.145020    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:49:46.162188 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:46.162123    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt" podStartSLOduration=1.162104172 podStartE2EDuration="1.162104172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:46.158978101 +0000 UTC m=+863.313056390" watchObservedRunningTime="2026-09-16 14:49:46.162104172 +0000 UTC m=+863.316182463"
Sep 16 14:49:46.180270 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:46.180220    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k" podStartSLOduration=1.180203964 podStartE2EDuration="1.180203964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:46.177458875 +0000 UTC m=+863.331537165" watchObservedRunningTime="2026-09-16 14:49:46.180203964 +0000 UTC m=+863.334282256"
Sep 16 14:49:47.086357 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:47.086323    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:49:47.086546 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:47.086411    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:49:48.092346 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:48.092313    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 14:49:54.986374 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:54.986313    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"]
Sep 16 14:49:54.990007 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:54.989988    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"
Sep 16 14:49:54.992466 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:54.992446    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 14:49:54.997676 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:54.997655    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"]
Sep 16 14:49:55.139879 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:55.139848    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:49:55.140067 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:55.139919    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:49:55.147987 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:55.147961    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9147f49-1332-435b-bd69-8c87aee8566c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx\" (UID: \"c9147f49-1332-435b-bd69-8c87aee8566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"
Sep 16 14:49:55.249289 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:55.249200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9147f49-1332-435b-bd69-8c87aee8566c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx\" (UID: \"c9147f49-1332-435b-bd69-8c87aee8566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"
Sep 16 14:49:55.249812 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:55.249792    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9147f49-1332-435b-bd69-8c87aee8566c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx\" (UID: \"c9147f49-1332-435b-bd69-8c87aee8566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"
Sep 16 14:49:55.302617 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:55.302589    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"
Sep 16 14:49:55.571441 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:55.571415    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"]
Sep 16 14:49:55.573539 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:49:55.573513    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9147f49_1332_435b_bd69_8c87aee8566c.slice/crio-7ca722786ec0a534ac5fdb6fb725539821cc24457f9ede8293b5fb8c21820b42 WatchSource:0}: Error finding container 7ca722786ec0a534ac5fdb6fb725539821cc24457f9ede8293b5fb8c21820b42: Status 404 returned error can't find the container with id 7ca722786ec0a534ac5fdb6fb725539821cc24457f9ede8293b5fb8c21820b42
Sep 16 14:49:56.203241 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:56.203201    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx" event={"ID":"c9147f49-1332-435b-bd69-8c87aee8566c","Type":"ContainerStarted","Data":"7ca722786ec0a534ac5fdb6fb725539821cc24457f9ede8293b5fb8c21820b42"}
Sep 16 14:49:57.154332 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:57.154301    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:49:57.154524 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:57.154447    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:49:59.223021 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:59.222928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx" event={"ID":"c9147f49-1332-435b-bd69-8c87aee8566c","Type":"ContainerStarted","Data":"57ea95c814c79d06faaf00086d0bc3d9925a96e0490d47c68a5f63d43f47cd66"}
Sep 16 14:49:59.242184 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:49:59.242120    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx" podStartSLOduration=2.169564893 podStartE2EDuration="5.242105225s" totalImagesPullingTime="3.072540332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:49:59.240366531 +0000 UTC m=+876.394444818" watchObservedRunningTime="2026-09-16 14:49:59.242105225 +0000 UTC m=+876.396183512"
Sep 16 14:50:00.228877 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:00.228842    2558 generic.go:356] "Generic (PLEG): container finished" podID="c9147f49-1332-435b-bd69-8c87aee8566c" containerID="57ea95c814c79d06faaf00086d0bc3d9925a96e0490d47c68a5f63d43f47cd66" exitCode=0
Sep 16 14:50:00.229285 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:00.228927    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx" event={"ID":"c9147f49-1332-435b-bd69-8c87aee8566c","Type":"ContainerDied","Data":"57ea95c814c79d06faaf00086d0bc3d9925a96e0490d47c68a5f63d43f47cd66"}
Sep 16 14:50:01.367953 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:01.367924    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"
Sep 16 14:50:01.510950 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:01.510870    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9147f49-1332-435b-bd69-8c87aee8566c-maas-api-service-ca\") pod \"c9147f49-1332-435b-bd69-8c87aee8566c\" (UID: \"c9147f49-1332-435b-bd69-8c87aee8566c\") "
Sep 16 14:50:01.511271 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:01.511247    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9147f49-1332-435b-bd69-8c87aee8566c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c9147f49-1332-435b-bd69-8c87aee8566c" (UID: "c9147f49-1332-435b-bd69-8c87aee8566c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:50:01.612344 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:01.612308    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9147f49-1332-435b-bd69-8c87aee8566c-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:50:02.241878 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:02.241784    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"
Sep 16 14:50:02.241878 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:02.241815    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx" event={"ID":"c9147f49-1332-435b-bd69-8c87aee8566c","Type":"ContainerDied","Data":"7ca722786ec0a534ac5fdb6fb725539821cc24457f9ede8293b5fb8c21820b42"}
Sep 16 14:50:02.241878 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:02.241861    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ca722786ec0a534ac5fdb6fb725539821cc24457f9ede8293b5fb8c21820b42"
Sep 16 14:50:04.691715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.691683    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"]
Sep 16 14:50:04.692254 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.692083    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9147f49-1332-435b-bd69-8c87aee8566c" containerName="revoke-keys"
Sep 16 14:50:04.692254 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.692093    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9147f49-1332-435b-bd69-8c87aee8566c" containerName="revoke-keys"
Sep 16 14:50:04.692254 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.692182    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9147f49-1332-435b-bd69-8c87aee8566c" containerName="revoke-keys"
Sep 16 14:50:04.695258 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.695239    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:50:04.698105 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.698080    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e689e5\""
Sep 16 14:50:04.698276 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.698257    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e689e5-dockercfg-krmlt\""
Sep 16 14:50:04.698351 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.698285    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"]
Sep 16 14:50:04.702366 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.701983    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:50:04.705141 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.705123    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"]
Sep 16 14:50:04.713947 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.713923    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"]
Sep 16 14:50:04.735294 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.735256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcs92\" (UniqueName: \"kubernetes.io/projected/4ac36623-d4f5-4b61-add2-6abf1b55118e-kube-api-access-hcs92\") pod \"payload-processing-e2e-inf-b-e689e5-5745678df9-5845n\" (UID: \"4ac36623-d4f5-4b61-add2-6abf1b55118e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:50:04.735487 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.735375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4znbh\" (UniqueName: \"kubernetes.io/projected/a8053eff-d86e-41f2-a928-b347d12f5034-kube-api-access-4znbh\") pod \"payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7\" (UID: \"a8053eff-d86e-41f2-a928-b347d12f5034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:50:04.735553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.735473    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8053eff-d86e-41f2-a928-b347d12f5034-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7\" (UID: \"a8053eff-d86e-41f2-a928-b347d12f5034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:50:04.735603 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.735571    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac36623-d4f5-4b61-add2-6abf1b55118e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e689e5-5745678df9-5845n\" (UID: \"4ac36623-d4f5-4b61-add2-6abf1b55118e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:50:04.836398 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.836362    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8053eff-d86e-41f2-a928-b347d12f5034-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7\" (UID: \"a8053eff-d86e-41f2-a928-b347d12f5034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:50:04.836567 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.836411    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac36623-d4f5-4b61-add2-6abf1b55118e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e689e5-5745678df9-5845n\" (UID: \"4ac36623-d4f5-4b61-add2-6abf1b55118e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:50:04.836567 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.836454    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcs92\" (UniqueName: \"kubernetes.io/projected/4ac36623-d4f5-4b61-add2-6abf1b55118e-kube-api-access-hcs92\") pod \"payload-processing-e2e-inf-b-e689e5-5745678df9-5845n\" (UID: \"4ac36623-d4f5-4b61-add2-6abf1b55118e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:50:04.836567 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.836496    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znbh\" (UniqueName: \"kubernetes.io/projected/a8053eff-d86e-41f2-a928-b347d12f5034-kube-api-access-4znbh\") pod \"payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7\" (UID: \"a8053eff-d86e-41f2-a928-b347d12f5034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:50:04.840201 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.837633    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac36623-d4f5-4b61-add2-6abf1b55118e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e689e5-5745678df9-5845n\" (UID: \"4ac36623-d4f5-4b61-add2-6abf1b55118e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:50:04.840201 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.837676    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8053eff-d86e-41f2-a928-b347d12f5034-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7\" (UID: \"a8053eff-d86e-41f2-a928-b347d12f5034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:50:04.845643 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.845620    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znbh\" (UniqueName: \"kubernetes.io/projected/a8053eff-d86e-41f2-a928-b347d12f5034-kube-api-access-4znbh\") pod \"payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7\" (UID: \"a8053eff-d86e-41f2-a928-b347d12f5034\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:50:04.845754 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:04.845710    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcs92\" (UniqueName: \"kubernetes.io/projected/4ac36623-d4f5-4b61-add2-6abf1b55118e-kube-api-access-hcs92\") pod \"payload-processing-e2e-inf-b-e689e5-5745678df9-5845n\" (UID: \"4ac36623-d4f5-4b61-add2-6abf1b55118e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:50:05.006513 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:05.006430    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:50:05.015516 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:05.015497    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:50:05.280580 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:05.280549    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"]
Sep 16 14:50:05.282601 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:50:05.282573    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8053eff_d86e_41f2_a928_b347d12f5034.slice/crio-3acf1079cf92efeb972837d2b5630c50e2b3f6de47bb7b4d3317fe780bd1327a WatchSource:0}: Error finding container 3acf1079cf92efeb972837d2b5630c50e2b3f6de47bb7b4d3317fe780bd1327a: Status 404 returned error can't find the container with id 3acf1079cf92efeb972837d2b5630c50e2b3f6de47bb7b4d3317fe780bd1327a
Sep 16 14:50:05.300681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:05.300659    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"]
Sep 16 14:50:05.302205 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:50:05.302179    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac36623_d4f5_4b61_add2_6abf1b55118e.slice/crio-e9094ef78dcb4f2a01dad0ea2dded2080e26cc3b58590806938fecb7d8851df9 WatchSource:0}: Error finding container e9094ef78dcb4f2a01dad0ea2dded2080e26cc3b58590806938fecb7d8851df9: Status 404 returned error can't find the container with id e9094ef78dcb4f2a01dad0ea2dded2080e26cc3b58590806938fecb7d8851df9
Sep 16 14:50:06.267077 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:06.267033    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n" event={"ID":"4ac36623-d4f5-4b61-add2-6abf1b55118e","Type":"ContainerStarted","Data":"5c71afd255b8aee652fec1587d22476dc710757813fc9bafeb66c3c129abdc7e"}
Sep 16 14:50:06.267499 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:06.267105    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n" event={"ID":"4ac36623-d4f5-4b61-add2-6abf1b55118e","Type":"ContainerStarted","Data":"e9094ef78dcb4f2a01dad0ea2dded2080e26cc3b58590806938fecb7d8851df9"}
Sep 16 14:50:06.267926 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:06.267907    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:50:06.269910 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:06.269880    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7" event={"ID":"a8053eff-d86e-41f2-a928-b347d12f5034","Type":"ContainerStarted","Data":"f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2"}
Sep 16 14:50:06.270065 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:06.269931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7" event={"ID":"a8053eff-d86e-41f2-a928-b347d12f5034","Type":"ContainerStarted","Data":"3acf1079cf92efeb972837d2b5630c50e2b3f6de47bb7b4d3317fe780bd1327a"}
Sep 16 14:50:06.270065 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:06.269961    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:50:06.285385 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:06.285330    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n" podStartSLOduration=2.28531515 podStartE2EDuration="2.28531515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:50:06.283145057 +0000 UTC m=+883.437223345" watchObservedRunningTime="2026-09-16 14:50:06.28531515 +0000 UTC m=+883.439393427"
Sep 16 14:50:06.300559 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:06.300516    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7" podStartSLOduration=2.300499889 podStartE2EDuration="2.300499889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:50:06.298719995 +0000 UTC m=+883.452798303" watchObservedRunningTime="2026-09-16 14:50:06.300499889 +0000 UTC m=+883.454578178"
Sep 16 14:50:07.953049 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:07.953016    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"]
Sep 16 14:50:07.953473 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:07.953237    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82" podUID="0e4547e3-0496-4514-a490-d6688b1a412f" containerName="payload-processing" containerID="cri-o://e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d" gracePeriod=30
Sep 16 14:50:07.962331 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:07.961509    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"]
Sep 16 14:50:07.962331 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:07.961722    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f" podUID="697fd466-de3f-4e3b-85f1-85d9deeff0c8" containerName="payload-pre-processing" containerID="cri-o://590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7" gracePeriod=30
Sep 16 14:50:08.133679 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.133631    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"]
Sep 16 14:50:08.139673 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.139640    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0252b3a9-bjnhx"]
Sep 16 14:50:08.221048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.221026    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:50:08.249582 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.249560    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:50:08.267718 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.267696    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qx9t\" (UniqueName: \"kubernetes.io/projected/697fd466-de3f-4e3b-85f1-85d9deeff0c8-kube-api-access-7qx9t\") pod \"697fd466-de3f-4e3b-85f1-85d9deeff0c8\" (UID: \"697fd466-de3f-4e3b-85f1-85d9deeff0c8\") "
Sep 16 14:50:08.267850 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.267739    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q5jr\" (UniqueName: \"kubernetes.io/projected/0e4547e3-0496-4514-a490-d6688b1a412f-kube-api-access-2q5jr\") pod \"0e4547e3-0496-4514-a490-d6688b1a412f\" (UID: \"0e4547e3-0496-4514-a490-d6688b1a412f\") "
Sep 16 14:50:08.267850 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.267757    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4547e3-0496-4514-a490-d6688b1a412f-plugins-config-volume\") pod \"0e4547e3-0496-4514-a490-d6688b1a412f\" (UID: \"0e4547e3-0496-4514-a490-d6688b1a412f\") "
Sep 16 14:50:08.267985 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.267891    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697fd466-de3f-4e3b-85f1-85d9deeff0c8-plugins-config-volume\") pod \"697fd466-de3f-4e3b-85f1-85d9deeff0c8\" (UID: \"697fd466-de3f-4e3b-85f1-85d9deeff0c8\") "
Sep 16 14:50:08.268243 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.268212    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e4547e3-0496-4514-a490-d6688b1a412f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e4547e3-0496-4514-a490-d6688b1a412f" (UID: "0e4547e3-0496-4514-a490-d6688b1a412f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:50:08.268335 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.268259    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/697fd466-de3f-4e3b-85f1-85d9deeff0c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "697fd466-de3f-4e3b-85f1-85d9deeff0c8" (UID: "697fd466-de3f-4e3b-85f1-85d9deeff0c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:50:08.269969 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.269945    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4547e3-0496-4514-a490-d6688b1a412f-kube-api-access-2q5jr" (OuterVolumeSpecName: "kube-api-access-2q5jr") pod "0e4547e3-0496-4514-a490-d6688b1a412f" (UID: "0e4547e3-0496-4514-a490-d6688b1a412f"). InnerVolumeSpecName "kube-api-access-2q5jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:50:08.270032 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.269949    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697fd466-de3f-4e3b-85f1-85d9deeff0c8-kube-api-access-7qx9t" (OuterVolumeSpecName: "kube-api-access-7qx9t") pod "697fd466-de3f-4e3b-85f1-85d9deeff0c8" (UID: "697fd466-de3f-4e3b-85f1-85d9deeff0c8"). InnerVolumeSpecName "kube-api-access-7qx9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:50:08.285314 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.285280    2558 generic.go:356] "Generic (PLEG): container finished" podID="697fd466-de3f-4e3b-85f1-85d9deeff0c8" containerID="590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7" exitCode=0
Sep 16 14:50:08.285462 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.285324    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"
Sep 16 14:50:08.285462 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.285355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f" event={"ID":"697fd466-de3f-4e3b-85f1-85d9deeff0c8","Type":"ContainerDied","Data":"590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7"}
Sep 16 14:50:08.285462 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.285428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f" event={"ID":"697fd466-de3f-4e3b-85f1-85d9deeff0c8","Type":"ContainerDied","Data":"4af1bb594e4156d0825551a76bbaca4e1f863ea99a76c39bd5378e911e1ba33e"}
Sep 16 14:50:08.285462 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.285454    2558 scope.go:117] "RemoveContainer" containerID="590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7"
Sep 16 14:50:08.287017 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.286996    2558 generic.go:356] "Generic (PLEG): container finished" podID="0e4547e3-0496-4514-a490-d6688b1a412f" containerID="e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d" exitCode=0
Sep 16 14:50:08.287120 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.287064    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"
Sep 16 14:50:08.287120 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.287110    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82" event={"ID":"0e4547e3-0496-4514-a490-d6688b1a412f","Type":"ContainerDied","Data":"e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d"}
Sep 16 14:50:08.287336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.287152    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82" event={"ID":"0e4547e3-0496-4514-a490-d6688b1a412f","Type":"ContainerDied","Data":"b739ddf453b010d874a918a96571948b63fa0030e154703a8842239d899f87f1"}
Sep 16 14:50:08.300310 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.300293    2558 scope.go:117] "RemoveContainer" containerID="590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7"
Sep 16 14:50:08.300636 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:50:08.300617    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7\": container with ID starting with 590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7 not found: ID does not exist" containerID="590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7"
Sep 16 14:50:08.300722 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.300643    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7"} err="failed to get container status \"590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7\": rpc error: code = NotFound desc = could not find container \"590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7\": container with ID starting with 590a098a7812b010a5727b099ae58a7a6c220bbe81edeb0b42643483fa88d7b7 not found: ID does not exist"
Sep 16 14:50:08.300722 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.300660    2558 scope.go:117] "RemoveContainer" containerID="e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d"
Sep 16 14:50:08.312850 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.312833    2558 scope.go:117] "RemoveContainer" containerID="e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d"
Sep 16 14:50:08.313142 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:50:08.313124    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d\": container with ID starting with e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d not found: ID does not exist" containerID="e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d"
Sep 16 14:50:08.313243 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.313147    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d"} err="failed to get container status \"e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d\": rpc error: code = NotFound desc = could not find container \"e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d\": container with ID starting with e9e5694af0d80e316146c067f08dfb1f72edcdcf98e36b8818ddedaefa991c5d not found: ID does not exist"
Sep 16 14:50:08.328698 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.328666    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"]
Sep 16 14:50:08.332108 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.332076    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0252b3a9-fbbc5fc99-4wm82"]
Sep 16 14:50:08.347564 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.347539    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"]
Sep 16 14:50:08.352266 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.352246    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0252b3a9-844c6cb4d6-5vz4f"]
Sep 16 14:50:08.368833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.368796    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q5jr\" (UniqueName: \"kubernetes.io/projected/0e4547e3-0496-4514-a490-d6688b1a412f-kube-api-access-2q5jr\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:50:08.368833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.368819    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e4547e3-0496-4514-a490-d6688b1a412f-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:50:08.368833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.368830    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697fd466-de3f-4e3b-85f1-85d9deeff0c8-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:50:08.368833 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:08.368838    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qx9t\" (UniqueName: \"kubernetes.io/projected/697fd466-de3f-4e3b-85f1-85d9deeff0c8-kube-api-access-7qx9t\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:50:09.295808 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:09.295731    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e4547e3-0496-4514-a490-d6688b1a412f" path="/var/lib/kubelet/pods/0e4547e3-0496-4514-a490-d6688b1a412f/volumes"
Sep 16 14:50:09.296144 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:09.296079    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="697fd466-de3f-4e3b-85f1-85d9deeff0c8" path="/var/lib/kubelet/pods/697fd466-de3f-4e3b-85f1-85d9deeff0c8/volumes"
Sep 16 14:50:09.296382 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:09.296370    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9147f49-1332-435b-bd69-8c87aee8566c" path="/var/lib/kubelet/pods/c9147f49-1332-435b-bd69-8c87aee8566c/volumes"
Sep 16 14:50:17.277808 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:17.277737    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:50:18.290068 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:18.290039    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:50:19.586235 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.586206    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"]
Sep 16 14:50:19.586634 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.586622    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="697fd466-de3f-4e3b-85f1-85d9deeff0c8" containerName="payload-pre-processing"
Sep 16 14:50:19.586634 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.586635    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="697fd466-de3f-4e3b-85f1-85d9deeff0c8" containerName="payload-pre-processing"
Sep 16 14:50:19.586714 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.586651    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e4547e3-0496-4514-a490-d6688b1a412f" containerName="payload-processing"
Sep 16 14:50:19.586714 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.586656    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e4547e3-0496-4514-a490-d6688b1a412f" containerName="payload-processing"
Sep 16 14:50:19.586777 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.586727    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e4547e3-0496-4514-a490-d6688b1a412f" containerName="payload-processing"
Sep 16 14:50:19.586777 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.586734    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="697fd466-de3f-4e3b-85f1-85d9deeff0c8" containerName="payload-pre-processing"
Sep 16 14:50:19.590891 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.590869    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:19.597102 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.597082    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8dd97f83-dockercfg-q7gwp\""
Sep 16 14:50:19.597256 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.597133    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8dd97f83\""
Sep 16 14:50:19.606285 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.606252    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"]
Sep 16 14:50:19.673408 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.673378    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwnhh\" (UniqueName: \"kubernetes.io/projected/cc2be253-6a45-49e2-bf07-2e72bb642d1f-kube-api-access-rwnhh\") pod \"payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt\" (UID: \"cc2be253-6a45-49e2-bf07-2e72bb642d1f\") " pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:19.673570 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.673491    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2be253-6a45-49e2-bf07-2e72bb642d1f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt\" (UID: \"cc2be253-6a45-49e2-bf07-2e72bb642d1f\") " pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:19.774411 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.774376    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwnhh\" (UniqueName: \"kubernetes.io/projected/cc2be253-6a45-49e2-bf07-2e72bb642d1f-kube-api-access-rwnhh\") pod \"payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt\" (UID: \"cc2be253-6a45-49e2-bf07-2e72bb642d1f\") " pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:19.774576 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.774466    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2be253-6a45-49e2-bf07-2e72bb642d1f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt\" (UID: \"cc2be253-6a45-49e2-bf07-2e72bb642d1f\") " pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:19.775141 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.775123    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2be253-6a45-49e2-bf07-2e72bb642d1f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt\" (UID: \"cc2be253-6a45-49e2-bf07-2e72bb642d1f\") " pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:19.783038 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.783009    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwnhh\" (UniqueName: \"kubernetes.io/projected/cc2be253-6a45-49e2-bf07-2e72bb642d1f-kube-api-access-rwnhh\") pod \"payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt\" (UID: \"cc2be253-6a45-49e2-bf07-2e72bb642d1f\") " pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:19.901334 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:19.901263    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:20.170790 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:20.170766    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"]
Sep 16 14:50:20.172394 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:50:20.172355    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2be253_6a45_49e2_bf07_2e72bb642d1f.slice/crio-981af2ccf416ffb159717cac9f1abda66c1edbc73be938e9220fd18187a0ac09 WatchSource:0}: Error finding container 981af2ccf416ffb159717cac9f1abda66c1edbc73be938e9220fd18187a0ac09: Status 404 returned error can't find the container with id 981af2ccf416ffb159717cac9f1abda66c1edbc73be938e9220fd18187a0ac09
Sep 16 14:50:20.354039 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:20.354003    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt" event={"ID":"cc2be253-6a45-49e2-bf07-2e72bb642d1f","Type":"ContainerStarted","Data":"f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f"}
Sep 16 14:50:20.354256 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:20.354053    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt" event={"ID":"cc2be253-6a45-49e2-bf07-2e72bb642d1f","Type":"ContainerStarted","Data":"981af2ccf416ffb159717cac9f1abda66c1edbc73be938e9220fd18187a0ac09"}
Sep 16 14:50:20.354256 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:20.354106    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:20.373867 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:20.373813    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt" podStartSLOduration=1.373796108 podStartE2EDuration="1.373796108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:50:20.373036621 +0000 UTC m=+897.527114909" watchObservedRunningTime="2026-09-16 14:50:20.373796108 +0000 UTC m=+897.527874402"
Sep 16 14:50:23.227268 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:23.227239    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:50:23.227712 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:23.227474    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:50:31.361965 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:31.361937    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:44.416774 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:44.416740    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"]
Sep 16 14:50:44.423953 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:44.423925    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"
Sep 16 14:50:44.426481 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:44.426460    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 14:50:44.431535 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:44.431513    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"]
Sep 16 14:50:44.608336 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:44.608304    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z\" (UID: \"ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"
Sep 16 14:50:44.709911 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:44.709830    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z\" (UID: \"ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"
Sep 16 14:50:44.710532 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:44.710511    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z\" (UID: \"ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"
Sep 16 14:50:44.736794 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:44.736767    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"
Sep 16 14:50:45.010120 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:45.010098    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"]
Sep 16 14:50:45.012241 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:50:45.012210    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4dfd58_ba6a_4d3f_9cb4_025a87c8a02f.slice/crio-70d7eda8ad172b43dbbaa8c24282765449b7ef948b8a27b7324c42efcc6bbd8b WatchSource:0}: Error finding container 70d7eda8ad172b43dbbaa8c24282765449b7ef948b8a27b7324c42efcc6bbd8b: Status 404 returned error can't find the container with id 70d7eda8ad172b43dbbaa8c24282765449b7ef948b8a27b7324c42efcc6bbd8b
Sep 16 14:50:45.480366 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:45.480330    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z" event={"ID":"ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f","Type":"ContainerStarted","Data":"de2c963b6763092f68376a61abdebdb9d0a288230a2759bf3673ba65c8384b7f"}
Sep 16 14:50:45.480738 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:45.480390    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z" event={"ID":"ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f","Type":"ContainerStarted","Data":"70d7eda8ad172b43dbbaa8c24282765449b7ef948b8a27b7324c42efcc6bbd8b"}
Sep 16 14:50:45.496596 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:45.496542    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z" podStartSLOduration=1.4965047089999999 podStartE2EDuration="1.496504709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:50:45.495326364 +0000 UTC m=+922.649404644" watchObservedRunningTime="2026-09-16 14:50:45.496504709 +0000 UTC m=+922.650582996"
Sep 16 14:50:46.487491 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:46.487456    2558 generic.go:356] "Generic (PLEG): container finished" podID="ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f" containerID="de2c963b6763092f68376a61abdebdb9d0a288230a2759bf3673ba65c8384b7f" exitCode=0
Sep 16 14:50:46.487864 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:46.487544    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z" event={"ID":"ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f","Type":"ContainerDied","Data":"de2c963b6763092f68376a61abdebdb9d0a288230a2759bf3673ba65c8384b7f"}
Sep 16 14:50:47.639951 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:47.639921    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"
Sep 16 14:50:47.737475 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:47.737436    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f-maas-api-service-ca\") pod \"ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f\" (UID: \"ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f\") "
Sep 16 14:50:47.737758 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:47.737737    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f" (UID: "ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:50:47.838200 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:47.838148    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:50:48.499097 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:48.499065    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"
Sep 16 14:50:48.499296 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:48.499102    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z" event={"ID":"ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f","Type":"ContainerDied","Data":"70d7eda8ad172b43dbbaa8c24282765449b7ef948b8a27b7324c42efcc6bbd8b"}
Sep 16 14:50:48.499296 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:48.499143    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70d7eda8ad172b43dbbaa8c24282765449b7ef948b8a27b7324c42efcc6bbd8b"
Sep 16 14:50:55.408306 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.408273    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"]
Sep 16 14:50:55.410762 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.408686    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f" containerName="revoke-keys"
Sep 16 14:50:55.410762 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.408697    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f" containerName="revoke-keys"
Sep 16 14:50:55.410762 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.408756    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f" containerName="revoke-keys"
Sep 16 14:50:55.411694 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.411675    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.415588 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.415567    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e33b3f\"/\"simulator-w4-e33b3f-kserve-self-signed-certs\""
Sep 16 14:50:55.415588 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.415580    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e33b3f\"/\"default-dockercfg-kg2wc\""
Sep 16 14:50:55.415753 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.415684    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e33b3f\"/\"openshift-service-ca.crt\""
Sep 16 14:50:55.415816 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.415792    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e33b3f\"/\"kube-root-ca.crt\""
Sep 16 14:50:55.422387 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.422366    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"]
Sep 16 14:50:55.510719 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.510686    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-dshm\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.510719 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.510721    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45jmm\" (UniqueName: \"kubernetes.io/projected/6c094eca-09ff-4eb2-88ca-17d4996793a9-kube-api-access-45jmm\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.510989 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.510745    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c094eca-09ff-4eb2-88ca-17d4996793a9-tls-certs\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.510989 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.510798    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-home\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.510989 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.510855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-tmp-dir\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.510989 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.510912    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-model-cache\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.611567 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.611521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-dshm\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.611567 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.611570    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45jmm\" (UniqueName: \"kubernetes.io/projected/6c094eca-09ff-4eb2-88ca-17d4996793a9-kube-api-access-45jmm\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.611840 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.611696    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c094eca-09ff-4eb2-88ca-17d4996793a9-tls-certs\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.611840 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.611750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-home\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.611840 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.611788    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-tmp-dir\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.611840 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.611833    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-model-cache\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.612139 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.612114    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-tmp-dir\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.612294 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.612224    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-model-cache\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.612294 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.612258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-home\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.613795 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.613765    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-dshm\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.614256 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.614237    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c094eca-09ff-4eb2-88ca-17d4996793a9-tls-certs\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.620346 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.620328    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45jmm\" (UniqueName: \"kubernetes.io/projected/6c094eca-09ff-4eb2-88ca-17d4996793a9-kube-api-access-45jmm\") pod \"simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") " pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.722701 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.722611    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:50:55.995040 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:55.995016    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"]
Sep 16 14:50:55.997243 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:50:55.997217    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c094eca_09ff_4eb2_88ca_17d4996793a9.slice/crio-cc17d9e67a8417ad0be6551a767d5688474be64d3ca798f02a1ba88464bd6cfc WatchSource:0}: Error finding container cc17d9e67a8417ad0be6551a767d5688474be64d3ca798f02a1ba88464bd6cfc: Status 404 returned error can't find the container with id cc17d9e67a8417ad0be6551a767d5688474be64d3ca798f02a1ba88464bd6cfc
Sep 16 14:50:56.091366 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.091328    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"]
Sep 16 14:50:56.091654 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.091603    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt" podUID="cc2be253-6a45-49e2-bf07-2e72bb642d1f" containerName="payload-processing" containerID="cri-o://f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f" gracePeriod=30
Sep 16 14:50:56.282441 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.282409    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"]
Sep 16 14:50:56.287923 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.287899    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8dd97f83-rq95z"]
Sep 16 14:50:56.349419 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.349396    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:56.421682 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.421651    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2be253-6a45-49e2-bf07-2e72bb642d1f-plugins-config-volume\") pod \"cc2be253-6a45-49e2-bf07-2e72bb642d1f\" (UID: \"cc2be253-6a45-49e2-bf07-2e72bb642d1f\") "
Sep 16 14:50:56.422154 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.421693    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwnhh\" (UniqueName: \"kubernetes.io/projected/cc2be253-6a45-49e2-bf07-2e72bb642d1f-kube-api-access-rwnhh\") pod \"cc2be253-6a45-49e2-bf07-2e72bb642d1f\" (UID: \"cc2be253-6a45-49e2-bf07-2e72bb642d1f\") "
Sep 16 14:50:56.422458 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.422432    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2be253-6a45-49e2-bf07-2e72bb642d1f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc2be253-6a45-49e2-bf07-2e72bb642d1f" (UID: "cc2be253-6a45-49e2-bf07-2e72bb642d1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:50:56.424101 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.424076    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2be253-6a45-49e2-bf07-2e72bb642d1f-kube-api-access-rwnhh" (OuterVolumeSpecName: "kube-api-access-rwnhh") pod "cc2be253-6a45-49e2-bf07-2e72bb642d1f" (UID: "cc2be253-6a45-49e2-bf07-2e72bb642d1f"). InnerVolumeSpecName "kube-api-access-rwnhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:50:56.522688 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.522601    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc2be253-6a45-49e2-bf07-2e72bb642d1f-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:50:56.522688 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.522632    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwnhh\" (UniqueName: \"kubernetes.io/projected/cc2be253-6a45-49e2-bf07-2e72bb642d1f-kube-api-access-rwnhh\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:50:56.540416 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.540382    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c" event={"ID":"6c094eca-09ff-4eb2-88ca-17d4996793a9","Type":"ContainerStarted","Data":"1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad"}
Sep 16 14:50:56.540416 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.540422    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c" event={"ID":"6c094eca-09ff-4eb2-88ca-17d4996793a9","Type":"ContainerStarted","Data":"cc17d9e67a8417ad0be6551a767d5688474be64d3ca798f02a1ba88464bd6cfc"}
Sep 16 14:50:56.541586 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.541564    2558 generic.go:356] "Generic (PLEG): container finished" podID="cc2be253-6a45-49e2-bf07-2e72bb642d1f" containerID="f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f" exitCode=0
Sep 16 14:50:56.541682 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.541599    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"
Sep 16 14:50:56.541682 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.541646    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt" event={"ID":"cc2be253-6a45-49e2-bf07-2e72bb642d1f","Type":"ContainerDied","Data":"f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f"}
Sep 16 14:50:56.541754 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.541689    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt" event={"ID":"cc2be253-6a45-49e2-bf07-2e72bb642d1f","Type":"ContainerDied","Data":"981af2ccf416ffb159717cac9f1abda66c1edbc73be938e9220fd18187a0ac09"}
Sep 16 14:50:56.541754 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.541714    2558 scope.go:117] "RemoveContainer" containerID="f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f"
Sep 16 14:50:56.550489 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.550470    2558 scope.go:117] "RemoveContainer" containerID="f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f"
Sep 16 14:50:56.550803 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:50:56.550784    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f\": container with ID starting with f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f not found: ID does not exist" containerID="f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f"
Sep 16 14:50:56.550902 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.550811    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f"} err="failed to get container status \"f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f\": rpc error: code = NotFound desc = could not find container \"f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f\": container with ID starting with f9b74f9013bbdbe7006aabb26254d1c913158f690bb8bf2c2e62b70fa4e1d77f not found: ID does not exist"
Sep 16 14:50:56.561345 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.561286    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c" podStartSLOduration=1.5612647480000001 podStartE2EDuration="1.561264748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:50:56.557886404 +0000 UTC m=+933.711964685" watchObservedRunningTime="2026-09-16 14:50:56.561264748 +0000 UTC m=+933.715343036"
Sep 16 14:50:56.573870 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.573842    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"]
Sep 16 14:50:56.579012 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:56.578988    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8dd97f83-6547bfd76b-pkmnt"]
Sep 16 14:50:57.295395 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:57.295360    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc2be253-6a45-49e2-bf07-2e72bb642d1f" path="/var/lib/kubelet/pods/cc2be253-6a45-49e2-bf07-2e72bb642d1f/volumes"
Sep 16 14:50:57.295705 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:50:57.295693    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f" path="/var/lib/kubelet/pods/ff4dfd58-ba6a-4d3f-9cb4-025a87c8a02f/volumes"
Sep 16 14:51:05.723756 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:05.723720    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:51:05.724141 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:05.723769    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:51:05.736191 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:05.736151    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:51:06.611185 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:06.611147    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 14:51:25.388362 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.388323    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"]
Sep 16 14:51:25.388840 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.388822    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2be253-6a45-49e2-bf07-2e72bb642d1f" containerName="payload-processing"
Sep 16 14:51:25.388888 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.388843    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2be253-6a45-49e2-bf07-2e72bb642d1f" containerName="payload-processing"
Sep 16 14:51:25.388923 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.388900    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2be253-6a45-49e2-bf07-2e72bb642d1f" containerName="payload-processing"
Sep 16 14:51:25.392993 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.392975    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"
Sep 16 14:51:25.395777 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.395758    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 14:51:25.399450 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.399421    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1e1ef-f474-4c13-8ad5-3649656e85e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p\" (UID: \"0fb1e1ef-f474-4c13-8ad5-3649656e85e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"
Sep 16 14:51:25.399664 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.399643    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"]
Sep 16 14:51:25.500621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.500584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1e1ef-f474-4c13-8ad5-3649656e85e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p\" (UID: \"0fb1e1ef-f474-4c13-8ad5-3649656e85e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"
Sep 16 14:51:25.501246 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.501223    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1e1ef-f474-4c13-8ad5-3649656e85e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p\" (UID: \"0fb1e1ef-f474-4c13-8ad5-3649656e85e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"
Sep 16 14:51:25.705089 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.705012    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"
Sep 16 14:51:25.974868 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:25.974790    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"]
Sep 16 14:51:25.978905 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:51:25.978883    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb1e1ef_f474_4c13_8ad5_3649656e85e6.slice/crio-4b71188adb551133f7ac43dce6c00bbab31d372e3d8a35c546eb7272d9ffab37 WatchSource:0}: Error finding container 4b71188adb551133f7ac43dce6c00bbab31d372e3d8a35c546eb7272d9ffab37: Status 404 returned error can't find the container with id 4b71188adb551133f7ac43dce6c00bbab31d372e3d8a35c546eb7272d9ffab37
Sep 16 14:51:26.708855 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:26.708826    2558 generic.go:356] "Generic (PLEG): container finished" podID="0fb1e1ef-f474-4c13-8ad5-3649656e85e6" containerID="6b37cfc5496f255226be2b77c5343a148c8c4e6dd2ecf950ba11dc50eb4a9ece" exitCode=0
Sep 16 14:51:26.709230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:26.708882    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p" event={"ID":"0fb1e1ef-f474-4c13-8ad5-3649656e85e6","Type":"ContainerDied","Data":"6b37cfc5496f255226be2b77c5343a148c8c4e6dd2ecf950ba11dc50eb4a9ece"}
Sep 16 14:51:26.709230 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:26.708924    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p" event={"ID":"0fb1e1ef-f474-4c13-8ad5-3649656e85e6","Type":"ContainerStarted","Data":"4b71188adb551133f7ac43dce6c00bbab31d372e3d8a35c546eb7272d9ffab37"}
Sep 16 14:51:27.855108 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:27.855085    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"
Sep 16 14:51:27.920780 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:27.920751    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1e1ef-f474-4c13-8ad5-3649656e85e6-maas-api-service-ca\") pod \"0fb1e1ef-f474-4c13-8ad5-3649656e85e6\" (UID: \"0fb1e1ef-f474-4c13-8ad5-3649656e85e6\") "
Sep 16 14:51:27.921200 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:27.921155    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb1e1ef-f474-4c13-8ad5-3649656e85e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0fb1e1ef-f474-4c13-8ad5-3649656e85e6" (UID: "0fb1e1ef-f474-4c13-8ad5-3649656e85e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:51:28.021584 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:28.021509    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1e1ef-f474-4c13-8ad5-3649656e85e6-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:51:28.720551 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:28.720520    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"
Sep 16 14:51:28.720738 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:28.720552    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p" event={"ID":"0fb1e1ef-f474-4c13-8ad5-3649656e85e6","Type":"ContainerDied","Data":"4b71188adb551133f7ac43dce6c00bbab31d372e3d8a35c546eb7272d9ffab37"}
Sep 16 14:51:28.720738 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:28.720597    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b71188adb551133f7ac43dce6c00bbab31d372e3d8a35c546eb7272d9ffab37"
Sep 16 14:51:32.018137 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:32.018089    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"]
Sep 16 14:51:32.021714 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:32.021686    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec2b25e3-96g5p"]
Sep 16 14:51:33.295756 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:33.295723    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fb1e1ef-f474-4c13-8ad5-3649656e85e6" path="/var/lib/kubelet/pods/0fb1e1ef-f474-4c13-8ad5-3649656e85e6/volumes"
Sep 16 14:51:42.397218 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.397179    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"]
Sep 16 14:51:42.397685 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.397672    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb1e1ef-f474-4c13-8ad5-3649656e85e6" containerName="revoke-keys"
Sep 16 14:51:42.397685 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.397686    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb1e1ef-f474-4c13-8ad5-3649656e85e6" containerName="revoke-keys"
Sep 16 14:51:42.397756 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.397748    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fb1e1ef-f474-4c13-8ad5-3649656e85e6" containerName="revoke-keys"
Sep 16 14:51:42.401084 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.401064    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"
Sep 16 14:51:42.404174 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.404140    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 14:51:42.413823 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.413797    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"]
Sep 16 14:51:42.457845 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.457817    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c466db4-6cdb-47be-bf1b-6433e95c4053-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bde531-fj7zm\" (UID: \"1c466db4-6cdb-47be-bf1b-6433e95c4053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"
Sep 16 14:51:42.558975 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.558934    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c466db4-6cdb-47be-bf1b-6433e95c4053-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bde531-fj7zm\" (UID: \"1c466db4-6cdb-47be-bf1b-6433e95c4053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"
Sep 16 14:51:42.559564 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.559542    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c466db4-6cdb-47be-bf1b-6433e95c4053-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-bde531-fj7zm\" (UID: \"1c466db4-6cdb-47be-bf1b-6433e95c4053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"
Sep 16 14:51:42.711562 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.711482    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"
Sep 16 14:51:42.983578 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:42.983548    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"]
Sep 16 14:51:42.985356 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:51:42.985331    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c466db4_6cdb_47be_bf1b_6433e95c4053.slice/crio-271b56c9ea0c26649c8f297f618ece88ad200fc56899c2f529b5fe13db00aacb WatchSource:0}: Error finding container 271b56c9ea0c26649c8f297f618ece88ad200fc56899c2f529b5fe13db00aacb: Status 404 returned error can't find the container with id 271b56c9ea0c26649c8f297f618ece88ad200fc56899c2f529b5fe13db00aacb
Sep 16 14:51:43.807934 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:43.807904    2558 generic.go:356] "Generic (PLEG): container finished" podID="1c466db4-6cdb-47be-bf1b-6433e95c4053" containerID="120c30939dbbe8932b1ec3e8f34c31c005ea2e73cae6b46a66876a9964a694b4" exitCode=0
Sep 16 14:51:43.808341 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:43.807979    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm" event={"ID":"1c466db4-6cdb-47be-bf1b-6433e95c4053","Type":"ContainerDied","Data":"120c30939dbbe8932b1ec3e8f34c31c005ea2e73cae6b46a66876a9964a694b4"}
Sep 16 14:51:43.808341 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:43.808015    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm" event={"ID":"1c466db4-6cdb-47be-bf1b-6433e95c4053","Type":"ContainerStarted","Data":"271b56c9ea0c26649c8f297f618ece88ad200fc56899c2f529b5fe13db00aacb"}
Sep 16 14:51:44.954824 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:44.954792    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"
Sep 16 14:51:45.085456 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:45.085367    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c466db4-6cdb-47be-bf1b-6433e95c4053-maas-api-service-ca\") pod \"1c466db4-6cdb-47be-bf1b-6433e95c4053\" (UID: \"1c466db4-6cdb-47be-bf1b-6433e95c4053\") "
Sep 16 14:51:45.085768 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:45.085742    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c466db4-6cdb-47be-bf1b-6433e95c4053-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c466db4-6cdb-47be-bf1b-6433e95c4053" (UID: "1c466db4-6cdb-47be-bf1b-6433e95c4053"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:51:45.186414 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:45.186384    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c466db4-6cdb-47be-bf1b-6433e95c4053-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:51:45.819805 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:45.819770    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm" event={"ID":"1c466db4-6cdb-47be-bf1b-6433e95c4053","Type":"ContainerDied","Data":"271b56c9ea0c26649c8f297f618ece88ad200fc56899c2f529b5fe13db00aacb"}
Sep 16 14:51:45.819805 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:45.819786    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"
Sep 16 14:51:45.820026 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:45.819818    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="271b56c9ea0c26649c8f297f618ece88ad200fc56899c2f529b5fe13db00aacb"
Sep 16 14:51:53.429024 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.428989    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"]
Sep 16 14:51:53.429508 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.429419    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k" podUID="ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb" containerName="payload-processing" containerID="cri-o://f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c" gracePeriod=30
Sep 16 14:51:53.434215 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.433936    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"]
Sep 16 14:51:53.434215 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.434136    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt" podUID="4cb2ec3d-b596-4eb7-8e63-66d45773bae7" containerName="payload-pre-processing" containerID="cri-o://161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd" gracePeriod=30
Sep 16 14:51:53.607488 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.607405    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"]
Sep 16 14:51:53.610492 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.610459    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-bde531-fj7zm"]
Sep 16 14:51:53.720641 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.720614    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:51:53.722697 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.722680    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:51:53.870686 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.870656    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgfjc\" (UniqueName: \"kubernetes.io/projected/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-kube-api-access-dgfjc\") pod \"4cb2ec3d-b596-4eb7-8e63-66d45773bae7\" (UID: \"4cb2ec3d-b596-4eb7-8e63-66d45773bae7\") "
Sep 16 14:51:53.870898 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.870707    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-plugins-config-volume\") pod \"4cb2ec3d-b596-4eb7-8e63-66d45773bae7\" (UID: \"4cb2ec3d-b596-4eb7-8e63-66d45773bae7\") "
Sep 16 14:51:53.870898 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.870767    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv7zb\" (UniqueName: \"kubernetes.io/projected/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-kube-api-access-gv7zb\") pod \"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb\" (UID: \"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb\") "
Sep 16 14:51:53.870898 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.870791    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-plugins-config-volume\") pod \"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb\" (UID: \"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb\") "
Sep 16 14:51:53.870898 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.870825    2558 generic.go:356] "Generic (PLEG): container finished" podID="ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb" containerID="f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c" exitCode=0
Sep 16 14:51:53.871130 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.870996    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"
Sep 16 14:51:53.871130 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.871110    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cb2ec3d-b596-4eb7-8e63-66d45773bae7" (UID: "4cb2ec3d-b596-4eb7-8e63-66d45773bae7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:51:53.871275 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.871193    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k" event={"ID":"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb","Type":"ContainerDied","Data":"f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c"}
Sep 16 14:51:53.871275 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.871254    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k" event={"ID":"ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb","Type":"ContainerDied","Data":"0aefeeceed150f4bef9bd02e6fa899ad952adf7fce1a7355231fdc15ba3e2bc1"}
Sep 16 14:51:53.871372 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.871286    2558 scope.go:117] "RemoveContainer" containerID="f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c"
Sep 16 14:51:53.871424 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.871395    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb" (UID: "ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:51:53.872882 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.872853    2558 generic.go:356] "Generic (PLEG): container finished" podID="4cb2ec3d-b596-4eb7-8e63-66d45773bae7" containerID="161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd" exitCode=0
Sep 16 14:51:53.873053 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.873010    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt" event={"ID":"4cb2ec3d-b596-4eb7-8e63-66d45773bae7","Type":"ContainerDied","Data":"161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd"}
Sep 16 14:51:53.873125 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.873056    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt" event={"ID":"4cb2ec3d-b596-4eb7-8e63-66d45773bae7","Type":"ContainerDied","Data":"053f6561203c19f7aa3d8412910dcab86c9892365eaaa11be8388d0847767300"}
Sep 16 14:51:53.873194 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.873123    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"
Sep 16 14:51:53.873671 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.873649    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-kube-api-access-gv7zb" (OuterVolumeSpecName: "kube-api-access-gv7zb") pod "ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb" (UID: "ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb"). InnerVolumeSpecName "kube-api-access-gv7zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:51:53.873908 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.873894    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-kube-api-access-dgfjc" (OuterVolumeSpecName: "kube-api-access-dgfjc") pod "4cb2ec3d-b596-4eb7-8e63-66d45773bae7" (UID: "4cb2ec3d-b596-4eb7-8e63-66d45773bae7"). InnerVolumeSpecName "kube-api-access-dgfjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:51:53.886735 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.886717    2558 scope.go:117] "RemoveContainer" containerID="f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c"
Sep 16 14:51:53.887030 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:51:53.887008    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c\": container with ID starting with f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c not found: ID does not exist" containerID="f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c"
Sep 16 14:51:53.887085 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.887040    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c"} err="failed to get container status \"f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c\": rpc error: code = NotFound desc = could not find container \"f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c\": container with ID starting with f2e34525820c8114188edcc7beea85dacaa7ae78545bc667b966abedf2c3099c not found: ID does not exist"
Sep 16 14:51:53.887085 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.887056    2558 scope.go:117] "RemoveContainer" containerID="161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd"
Sep 16 14:51:53.896453 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.896437    2558 scope.go:117] "RemoveContainer" containerID="161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd"
Sep 16 14:51:53.896739 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:51:53.896721    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd\": container with ID starting with 161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd not found: ID does not exist" containerID="161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd"
Sep 16 14:51:53.896794 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.896749    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd"} err="failed to get container status \"161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd\": rpc error: code = NotFound desc = could not find container \"161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd\": container with ID starting with 161cbfd466fbc973b41148afe5fe9af6dbe30bdcd9bf4bc326fc92dbde9a65cd not found: ID does not exist"
Sep 16 14:51:53.978074 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.972736    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgfjc\" (UniqueName: \"kubernetes.io/projected/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-kube-api-access-dgfjc\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:51:53.978074 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.972808    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb2ec3d-b596-4eb7-8e63-66d45773bae7-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:51:53.978074 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.972833    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv7zb\" (UniqueName: \"kubernetes.io/projected/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-kube-api-access-gv7zb\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:51:53.978074 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:53.972849    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:51:54.194674 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:54.194647    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"]
Sep 16 14:51:54.201975 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:54.201953    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bde531-5ffc777dfc-nm92k"]
Sep 16 14:51:54.223109 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:54.223083    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"]
Sep 16 14:51:54.225376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:54.225354    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bde531-6bd8dfc894-ng8mt"]
Sep 16 14:51:55.295611 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:55.295583    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c466db4-6cdb-47be-bf1b-6433e95c4053" path="/var/lib/kubelet/pods/1c466db4-6cdb-47be-bf1b-6433e95c4053/volumes"
Sep 16 14:51:55.295977 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:55.295921    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cb2ec3d-b596-4eb7-8e63-66d45773bae7" path="/var/lib/kubelet/pods/4cb2ec3d-b596-4eb7-8e63-66d45773bae7/volumes"
Sep 16 14:51:55.296246 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:51:55.296234    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb" path="/var/lib/kubelet/pods/ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb/volumes"
Sep 16 14:52:01.548359 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.548325    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"]
Sep 16 14:52:01.548718 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.548701    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c466db4-6cdb-47be-bf1b-6433e95c4053" containerName="revoke-keys"
Sep 16 14:52:01.548718 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.548711    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c466db4-6cdb-47be-bf1b-6433e95c4053" containerName="revoke-keys"
Sep 16 14:52:01.548800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.548722    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb2ec3d-b596-4eb7-8e63-66d45773bae7" containerName="payload-pre-processing"
Sep 16 14:52:01.548800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.548727    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb2ec3d-b596-4eb7-8e63-66d45773bae7" containerName="payload-pre-processing"
Sep 16 14:52:01.548800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.548742    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb" containerName="payload-processing"
Sep 16 14:52:01.548800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.548747    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb" containerName="payload-processing"
Sep 16 14:52:01.548919 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.548810    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf7cd73-6e18-40da-9fa2-b40e0da7c8bb" containerName="payload-processing"
Sep 16 14:52:01.548919 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.548822    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c466db4-6cdb-47be-bf1b-6433e95c4053" containerName="revoke-keys"
Sep 16 14:52:01.548919 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.548828    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb2ec3d-b596-4eb7-8e63-66d45773bae7" containerName="payload-pre-processing"
Sep 16 14:52:01.553557 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.553539    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"
Sep 16 14:52:01.557255 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.557218    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 14:52:01.559076 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.559016    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"]
Sep 16 14:52:01.747582 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.747546    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea83a17-92f2-4dc1-97a8-12907f5a6c5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eb086ade-62np4\" (UID: \"7ea83a17-92f2-4dc1-97a8-12907f5a6c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"
Sep 16 14:52:01.848570 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.848539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea83a17-92f2-4dc1-97a8-12907f5a6c5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eb086ade-62np4\" (UID: \"7ea83a17-92f2-4dc1-97a8-12907f5a6c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"
Sep 16 14:52:01.849207 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.849158    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea83a17-92f2-4dc1-97a8-12907f5a6c5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-eb086ade-62np4\" (UID: \"7ea83a17-92f2-4dc1-97a8-12907f5a6c5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"
Sep 16 14:52:01.866636 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:01.866577    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"
Sep 16 14:52:02.137429 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:02.137391    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"]
Sep 16 14:52:02.139342 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:52:02.139314    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea83a17_92f2_4dc1_97a8_12907f5a6c5b.slice/crio-36763351b77f2cd96edbf548a1e6792ade8c5d168b60cf84a01170e3265a2dda WatchSource:0}: Error finding container 36763351b77f2cd96edbf548a1e6792ade8c5d168b60cf84a01170e3265a2dda: Status 404 returned error can't find the container with id 36763351b77f2cd96edbf548a1e6792ade8c5d168b60cf84a01170e3265a2dda
Sep 16 14:52:02.927210 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:02.927174    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4" event={"ID":"7ea83a17-92f2-4dc1-97a8-12907f5a6c5b","Type":"ContainerStarted","Data":"7eda4b306405fad3397167aaa8f98eb31df227d76f119dbb886b71b673d73c65"}
Sep 16 14:52:02.927591 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:02.927239    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4" event={"ID":"7ea83a17-92f2-4dc1-97a8-12907f5a6c5b","Type":"ContainerStarted","Data":"36763351b77f2cd96edbf548a1e6792ade8c5d168b60cf84a01170e3265a2dda"}
Sep 16 14:52:02.960110 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:02.960063    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4" podStartSLOduration=1.9600458889999999 podStartE2EDuration="1.960045889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:52:02.958271467 +0000 UTC m=+1000.112349756" watchObservedRunningTime="2026-09-16 14:52:02.960045889 +0000 UTC m=+1000.114124175"
Sep 16 14:52:03.934811 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:03.934774    2558 generic.go:356] "Generic (PLEG): container finished" podID="7ea83a17-92f2-4dc1-97a8-12907f5a6c5b" containerID="7eda4b306405fad3397167aaa8f98eb31df227d76f119dbb886b71b673d73c65" exitCode=0
Sep 16 14:52:03.935212 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:03.934859    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4" event={"ID":"7ea83a17-92f2-4dc1-97a8-12907f5a6c5b","Type":"ContainerDied","Data":"7eda4b306405fad3397167aaa8f98eb31df227d76f119dbb886b71b673d73c65"}
Sep 16 14:52:05.072560 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:05.072539    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"
Sep 16 14:52:05.078402 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:05.078386    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea83a17-92f2-4dc1-97a8-12907f5a6c5b-maas-api-service-ca\") pod \"7ea83a17-92f2-4dc1-97a8-12907f5a6c5b\" (UID: \"7ea83a17-92f2-4dc1-97a8-12907f5a6c5b\") "
Sep 16 14:52:05.078679 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:05.078660    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea83a17-92f2-4dc1-97a8-12907f5a6c5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ea83a17-92f2-4dc1-97a8-12907f5a6c5b" (UID: "7ea83a17-92f2-4dc1-97a8-12907f5a6c5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:52:05.179817 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:05.179776    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ea83a17-92f2-4dc1-97a8-12907f5a6c5b-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:52:05.945807 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:05.945764    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4" event={"ID":"7ea83a17-92f2-4dc1-97a8-12907f5a6c5b","Type":"ContainerDied","Data":"36763351b77f2cd96edbf548a1e6792ade8c5d168b60cf84a01170e3265a2dda"}
Sep 16 14:52:05.945807 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:05.945793    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"
Sep 16 14:52:05.945807 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:05.945817    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36763351b77f2cd96edbf548a1e6792ade8c5d168b60cf84a01170e3265a2dda"
Sep 16 14:52:08.922500 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:08.922469    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"]
Sep 16 14:52:08.924842 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:08.924821    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-eb086ade-62np4"]
Sep 16 14:52:09.295842 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:09.295754    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ea83a17-92f2-4dc1-97a8-12907f5a6c5b" path="/var/lib/kubelet/pods/7ea83a17-92f2-4dc1-97a8-12907f5a6c5b/volumes"
Sep 16 14:52:37.604322 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.604287    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"]
Sep 16 14:52:37.604777 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.604677    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ea83a17-92f2-4dc1-97a8-12907f5a6c5b" containerName="revoke-keys"
Sep 16 14:52:37.604777 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.604688    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ea83a17-92f2-4dc1-97a8-12907f5a6c5b" containerName="revoke-keys"
Sep 16 14:52:37.604777 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.604750    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ea83a17-92f2-4dc1-97a8-12907f5a6c5b" containerName="revoke-keys"
Sep 16 14:52:37.609385 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.609359    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.613388 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.613364    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-780e10\"/\"openshift-service-ca.crt\""
Sep 16 14:52:37.613529 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.613388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-780e10\"/\"premium-w0-780e10-kserve-self-signed-certs\""
Sep 16 14:52:37.613529 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.613365    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-780e10\"/\"kube-root-ca.crt\""
Sep 16 14:52:37.613529 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.613476    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-780e10\"/\"default-dockercfg-drkdz\""
Sep 16 14:52:37.619655 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.619631    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"]
Sep 16 14:52:37.787599 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.787564    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-dshm\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.787759 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.787644    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59040847-f4ee-4371-a882-10fc0a2d4661-tls-certs\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.787759 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.787672    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-home\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.787759 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.787685    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-tmp-dir\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.787869 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.787789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9flq\" (UniqueName: \"kubernetes.io/projected/59040847-f4ee-4371-a882-10fc0a2d4661-kube-api-access-q9flq\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.787869 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.787818    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-model-cache\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.889208 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.889120    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-dshm\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.889208 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.889199    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59040847-f4ee-4371-a882-10fc0a2d4661-tls-certs\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.889401 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.889229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-home\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.889401 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.889244    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-tmp-dir\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.889401 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.889275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9flq\" (UniqueName: \"kubernetes.io/projected/59040847-f4ee-4371-a882-10fc0a2d4661-kube-api-access-q9flq\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.889401 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.889306    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-model-cache\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.889724 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.889700    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-tmp-dir\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.889823 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.889706    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-home\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.889823 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.889740    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-model-cache\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.891784 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.891765    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-dshm\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.891841 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.891790    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59040847-f4ee-4371-a882-10fc0a2d4661-tls-certs\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.897488 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.897464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9flq\" (UniqueName: \"kubernetes.io/projected/59040847-f4ee-4371-a882-10fc0a2d4661-kube-api-access-q9flq\") pod \"premium-w0-780e10-kserve-56cc7b6bf5-st7l2\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") " pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:37.919425 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:37.919400    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:38.191010 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:38.190987    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"]
Sep 16 14:52:38.193290 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:52:38.193267    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59040847_f4ee_4371_a882_10fc0a2d4661.slice/crio-a09e8dbcb52fbc28cefeccff9f34a574caa64b6859d5552e87d147f48e79baf8 WatchSource:0}: Error finding container a09e8dbcb52fbc28cefeccff9f34a574caa64b6859d5552e87d147f48e79baf8: Status 404 returned error can't find the container with id a09e8dbcb52fbc28cefeccff9f34a574caa64b6859d5552e87d147f48e79baf8
Sep 16 14:52:39.119847 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:39.119810    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2" event={"ID":"59040847-f4ee-4371-a882-10fc0a2d4661","Type":"ContainerStarted","Data":"a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29"}
Sep 16 14:52:39.120222 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:39.119856    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2" event={"ID":"59040847-f4ee-4371-a882-10fc0a2d4661","Type":"ContainerStarted","Data":"a09e8dbcb52fbc28cefeccff9f34a574caa64b6859d5552e87d147f48e79baf8"}
Sep 16 14:52:39.139438 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:39.139389    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2" podStartSLOduration=2.139372771 podStartE2EDuration="2.139372771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:52:39.139061216 +0000 UTC m=+1036.293139504" watchObservedRunningTime="2026-09-16 14:52:39.139372771 +0000 UTC m=+1036.293451059"
Sep 16 14:52:47.920331 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:47.920287    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:47.920804 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:47.920350    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:47.932813 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:47.932783    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:48.177196 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:48.177094    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 14:52:55.905441 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:55.905407    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"]
Sep 16 14:52:55.908940 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:55.908922    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:55.911589 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:55.911568    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e33b3f\"/\"premium-w4-e33b3f-kserve-self-signed-certs\""
Sep 16 14:52:55.918110 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:55.918088    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"]
Sep 16 14:52:55.966727 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:55.966700    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-model-cache\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:55.966871 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:55.966730    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-dshm\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:55.966871 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:55.966751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tmp-dir\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:55.966871 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:55.966767    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-home\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:55.966871 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:55.966822    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svf4\" (UniqueName: \"kubernetes.io/projected/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-kube-api-access-7svf4\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:55.967035 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:55.966903    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tls-certs\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.068013 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.067973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-model-cache\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.068013 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.068017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-dshm\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.068294 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.068042    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tmp-dir\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.068294 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.068065    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-home\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.068294 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.068097    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svf4\" (UniqueName: \"kubernetes.io/projected/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-kube-api-access-7svf4\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.068294 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.068154    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tls-certs\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.068502 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.068384    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-model-cache\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.068502 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.068476    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tmp-dir\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.068591 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.068500    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-home\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.070282 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.070260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-dshm\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.070635 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.070615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tls-certs\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.077027 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.077004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svf4\" (UniqueName: \"kubernetes.io/projected/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-kube-api-access-7svf4\") pod \"premium-w4-e33b3f-kserve-688d464c5b-d7t4k\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") " pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.219674 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.219592    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:52:56.488597 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:56.488574    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"]
Sep 16 14:52:56.490736 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:52:56.490707    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod047ddc24_2ba8_4e95_b6f6_a3fa6c4a4b18.slice/crio-45304fa588c5bb5e5e4f4cba1097835c74f61e6f7c72cf9a285320484e3c8baf WatchSource:0}: Error finding container 45304fa588c5bb5e5e4f4cba1097835c74f61e6f7c72cf9a285320484e3c8baf: Status 404 returned error can't find the container with id 45304fa588c5bb5e5e4f4cba1097835c74f61e6f7c72cf9a285320484e3c8baf
Sep 16 14:52:57.213632 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:57.213599    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k" event={"ID":"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18","Type":"ContainerStarted","Data":"34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405"}
Sep 16 14:52:57.214023 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:57.213641    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k" event={"ID":"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18","Type":"ContainerStarted","Data":"45304fa588c5bb5e5e4f4cba1097835c74f61e6f7c72cf9a285320484e3c8baf"}
Sep 16 14:52:57.231451 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:52:57.231407    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k" podStartSLOduration=2.231391627 podStartE2EDuration="2.231391627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:52:57.230135871 +0000 UTC m=+1054.384214159" watchObservedRunningTime="2026-09-16 14:52:57.231391627 +0000 UTC m=+1054.385469959"
Sep 16 14:53:06.219958 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:06.219918    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:53:06.220393 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:06.219986    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:53:06.232138 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:06.232115    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:53:06.273405 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:06.273371    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 14:53:58.096178 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.096137    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"]
Sep 16 14:53:58.100072 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.100052    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:53:58.102858 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.102831    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a743146e\""
Sep 16 14:53:58.102979 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.102934    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a743146e-dockercfg-m5cvl\""
Sep 16 14:53:58.118794 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.118773    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"]
Sep 16 14:53:58.277133 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.277103    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fql52\" (UniqueName: \"kubernetes.io/projected/c02a848a-daaa-419b-a0e1-5e8c4493cd18-kube-api-access-fql52\") pod \"payload-processing-e2e-mt-a743146e-df6bb76f-pswb2\" (UID: \"c02a848a-daaa-419b-a0e1-5e8c4493cd18\") " pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:53:58.277318 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.277195    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02a848a-daaa-419b-a0e1-5e8c4493cd18-plugins-config-volume\") pod \"payload-processing-e2e-mt-a743146e-df6bb76f-pswb2\" (UID: \"c02a848a-daaa-419b-a0e1-5e8c4493cd18\") " pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:53:58.378323 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.378252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02a848a-daaa-419b-a0e1-5e8c4493cd18-plugins-config-volume\") pod \"payload-processing-e2e-mt-a743146e-df6bb76f-pswb2\" (UID: \"c02a848a-daaa-419b-a0e1-5e8c4493cd18\") " pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:53:58.378468 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.378343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fql52\" (UniqueName: \"kubernetes.io/projected/c02a848a-daaa-419b-a0e1-5e8c4493cd18-kube-api-access-fql52\") pod \"payload-processing-e2e-mt-a743146e-df6bb76f-pswb2\" (UID: \"c02a848a-daaa-419b-a0e1-5e8c4493cd18\") " pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:53:58.378935 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.378915    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02a848a-daaa-419b-a0e1-5e8c4493cd18-plugins-config-volume\") pod \"payload-processing-e2e-mt-a743146e-df6bb76f-pswb2\" (UID: \"c02a848a-daaa-419b-a0e1-5e8c4493cd18\") " pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:53:58.389298 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.389275    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fql52\" (UniqueName: \"kubernetes.io/projected/c02a848a-daaa-419b-a0e1-5e8c4493cd18-kube-api-access-fql52\") pod \"payload-processing-e2e-mt-a743146e-df6bb76f-pswb2\" (UID: \"c02a848a-daaa-419b-a0e1-5e8c4493cd18\") " pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:53:58.409964 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.409933    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:53:58.684212 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:58.684154    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"]
Sep 16 14:53:58.686643 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:53:58.686615    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc02a848a_daaa_419b_a0e1_5e8c4493cd18.slice/crio-7e2bd2d8abb1651b25ed72e47b44cbdafb5c422ee6e23627d6784cf1982b3006 WatchSource:0}: Error finding container 7e2bd2d8abb1651b25ed72e47b44cbdafb5c422ee6e23627d6784cf1982b3006: Status 404 returned error can't find the container with id 7e2bd2d8abb1651b25ed72e47b44cbdafb5c422ee6e23627d6784cf1982b3006
Sep 16 14:53:59.532540 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:59.532500    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2" event={"ID":"c02a848a-daaa-419b-a0e1-5e8c4493cd18","Type":"ContainerStarted","Data":"a9174cd978b8f355a19a9efdd8332308dcf54dfce3275e7eb18ebac1a968cd66"}
Sep 16 14:53:59.533036 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:59.532567    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2" event={"ID":"c02a848a-daaa-419b-a0e1-5e8c4493cd18","Type":"ContainerStarted","Data":"7e2bd2d8abb1651b25ed72e47b44cbdafb5c422ee6e23627d6784cf1982b3006"}
Sep 16 14:53:59.533036 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:59.532705    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:53:59.553563 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:53:59.553516    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2" podStartSLOduration=1.553501493 podStartE2EDuration="1.553501493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:53:59.55134142 +0000 UTC m=+1116.705419729" watchObservedRunningTime="2026-09-16 14:53:59.553501493 +0000 UTC m=+1116.707579780"
Sep 16 14:54:10.539592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.539563    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:54:10.619666 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.619634    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"]
Sep 16 14:54:10.623684 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.623661    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.626447 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.626426    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-230a34\"/\"explicit-model-230a34-kserve-self-signed-certs\""
Sep 16 14:54:10.626571 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.626445    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-230a34\"/\"openshift-service-ca.crt\""
Sep 16 14:54:10.626571 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.626494    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-230a34\"/\"kube-root-ca.crt\""
Sep 16 14:54:10.626752 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.626736    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-230a34\"/\"default-dockercfg-292lv\""
Sep 16 14:54:10.630995 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.630974    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"]
Sep 16 14:54:10.686589 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.686559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-tmp-dir\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.686749 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.686596    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gdbj\" (UniqueName: \"kubernetes.io/projected/78b53180-f8c7-46fa-b488-c19795c3374c-kube-api-access-5gdbj\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.686749 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.686619    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-model-cache\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.686841 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.686761    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-home\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.686841 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.686801    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-dshm\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.686841 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.686815    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b53180-f8c7-46fa-b488-c19795c3374c-tls-certs\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.787831 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.787798    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-home\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.788013 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.787851    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-dshm\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.788013 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.787877    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b53180-f8c7-46fa-b488-c19795c3374c-tls-certs\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.788013 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.787902    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-tmp-dir\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.788013 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.787937    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gdbj\" (UniqueName: \"kubernetes.io/projected/78b53180-f8c7-46fa-b488-c19795c3374c-kube-api-access-5gdbj\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.788013 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.787968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-model-cache\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.788370 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.788280    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-home\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.788447 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.788372    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-model-cache\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.788561 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.788526    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-tmp-dir\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.790697 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.790644    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-dshm\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.790819 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.790804    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b53180-f8c7-46fa-b488-c19795c3374c-tls-certs\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.797609 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.797590    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gdbj\" (UniqueName: \"kubernetes.io/projected/78b53180-f8c7-46fa-b488-c19795c3374c-kube-api-access-5gdbj\") pod \"explicit-model-230a34-kserve-779794d67f-blvbt\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") " pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:10.934521 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:10.934489    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:11.206665 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:11.206640    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"]
Sep 16 14:54:11.208398 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:54:11.208372    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78b53180_f8c7_46fa_b488_c19795c3374c.slice/crio-71c5d814f3a135c19b978c17fdc260e23166b0bfdc87fabb1e13462f732db9dd WatchSource:0}: Error finding container 71c5d814f3a135c19b978c17fdc260e23166b0bfdc87fabb1e13462f732db9dd: Status 404 returned error can't find the container with id 71c5d814f3a135c19b978c17fdc260e23166b0bfdc87fabb1e13462f732db9dd
Sep 16 14:54:11.601838 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:11.601800    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt" event={"ID":"78b53180-f8c7-46fa-b488-c19795c3374c","Type":"ContainerStarted","Data":"85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2"}
Sep 16 14:54:11.602250 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:11.601850    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt" event={"ID":"78b53180-f8c7-46fa-b488-c19795c3374c","Type":"ContainerStarted","Data":"71c5d814f3a135c19b978c17fdc260e23166b0bfdc87fabb1e13462f732db9dd"}
Sep 16 14:54:11.619829 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:11.619782    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt" podStartSLOduration=1.619766329 podStartE2EDuration="1.619766329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:54:11.618674412 +0000 UTC m=+1128.772752701" watchObservedRunningTime="2026-09-16 14:54:11.619766329 +0000 UTC m=+1128.773844679"
Sep 16 14:54:20.935222 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:20.935154    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:20.935769 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:20.935751    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:20.964576 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:20.964544    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:21.680344 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:21.680310    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:54:30.455631 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:30.455589    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"]
Sep 16 14:54:30.461225 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:30.461087    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"]
Sep 16 14:54:30.461225 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:30.461207    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"
Sep 16 14:54:30.465465 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:30.465428    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 14:54:30.592764 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:30.592731    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6df49-fee3-4d12-a7c0-024ed86862fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a743146e-8wqq6\" (UID: \"f1c6df49-fee3-4d12-a7c0-024ed86862fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"
Sep 16 14:54:30.693807 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:30.693777    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6df49-fee3-4d12-a7c0-024ed86862fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a743146e-8wqq6\" (UID: \"f1c6df49-fee3-4d12-a7c0-024ed86862fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"
Sep 16 14:54:30.694429 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:30.694405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6df49-fee3-4d12-a7c0-024ed86862fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-a743146e-8wqq6\" (UID: \"f1c6df49-fee3-4d12-a7c0-024ed86862fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"
Sep 16 14:54:30.774766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:30.774685    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"
Sep 16 14:54:31.064972 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:31.064938    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"]
Sep 16 14:54:31.090101 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:54:31.090073    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c6df49_fee3_4d12_a7c0_024ed86862fd.slice/crio-5b0291a34af1130f46aaeadbdc65154144fd0e4f55de92f5e3e85b65c82f7db3 WatchSource:0}: Error finding container 5b0291a34af1130f46aaeadbdc65154144fd0e4f55de92f5e3e85b65c82f7db3: Status 404 returned error can't find the container with id 5b0291a34af1130f46aaeadbdc65154144fd0e4f55de92f5e3e85b65c82f7db3
Sep 16 14:54:31.728589 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:31.728559    2558 generic.go:356] "Generic (PLEG): container finished" podID="f1c6df49-fee3-4d12-a7c0-024ed86862fd" containerID="b2600d940d560b3c13f41cbc865bd4af06a708864e113cab504efe0af7290741" exitCode=0
Sep 16 14:54:31.728960 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:31.728613    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6" event={"ID":"f1c6df49-fee3-4d12-a7c0-024ed86862fd","Type":"ContainerDied","Data":"b2600d940d560b3c13f41cbc865bd4af06a708864e113cab504efe0af7290741"}
Sep 16 14:54:31.728960 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:31.728654    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6" event={"ID":"f1c6df49-fee3-4d12-a7c0-024ed86862fd","Type":"ContainerStarted","Data":"5b0291a34af1130f46aaeadbdc65154144fd0e4f55de92f5e3e85b65c82f7db3"}
Sep 16 14:54:32.858476 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:32.858453    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"
Sep 16 14:54:33.017711 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:33.017621    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6df49-fee3-4d12-a7c0-024ed86862fd-maas-api-service-ca\") pod \"f1c6df49-fee3-4d12-a7c0-024ed86862fd\" (UID: \"f1c6df49-fee3-4d12-a7c0-024ed86862fd\") "
Sep 16 14:54:33.018019 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:33.017993    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c6df49-fee3-4d12-a7c0-024ed86862fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1c6df49-fee3-4d12-a7c0-024ed86862fd" (UID: "f1c6df49-fee3-4d12-a7c0-024ed86862fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:54:33.118448 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:33.118404    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c6df49-fee3-4d12-a7c0-024ed86862fd-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:54:33.740659 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:33.740586    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"
Sep 16 14:54:33.740806 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:33.740585    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6" event={"ID":"f1c6df49-fee3-4d12-a7c0-024ed86862fd","Type":"ContainerDied","Data":"5b0291a34af1130f46aaeadbdc65154144fd0e4f55de92f5e3e85b65c82f7db3"}
Sep 16 14:54:33.740806 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:33.740704    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b0291a34af1130f46aaeadbdc65154144fd0e4f55de92f5e3e85b65c82f7db3"
Sep 16 14:54:41.557263 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:41.557222    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"]
Sep 16 14:54:41.557792 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:41.557467    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2" podUID="c02a848a-daaa-419b-a0e1-5e8c4493cd18" containerName="payload-processing" containerID="cri-o://a9174cd978b8f355a19a9efdd8332308dcf54dfce3275e7eb18ebac1a968cd66" gracePeriod=30
Sep 16 14:54:41.785889 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:41.785851    2558 generic.go:356] "Generic (PLEG): container finished" podID="c02a848a-daaa-419b-a0e1-5e8c4493cd18" containerID="a9174cd978b8f355a19a9efdd8332308dcf54dfce3275e7eb18ebac1a968cd66" exitCode=0
Sep 16 14:54:41.786077 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:41.785914    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2" event={"ID":"c02a848a-daaa-419b-a0e1-5e8c4493cd18","Type":"ContainerDied","Data":"a9174cd978b8f355a19a9efdd8332308dcf54dfce3275e7eb18ebac1a968cd66"}
Sep 16 14:54:41.823594 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:41.823517    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:54:41.903557 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:41.903508    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fql52\" (UniqueName: \"kubernetes.io/projected/c02a848a-daaa-419b-a0e1-5e8c4493cd18-kube-api-access-fql52\") pod \"c02a848a-daaa-419b-a0e1-5e8c4493cd18\" (UID: \"c02a848a-daaa-419b-a0e1-5e8c4493cd18\") "
Sep 16 14:54:41.903741 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:41.903610    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02a848a-daaa-419b-a0e1-5e8c4493cd18-plugins-config-volume\") pod \"c02a848a-daaa-419b-a0e1-5e8c4493cd18\" (UID: \"c02a848a-daaa-419b-a0e1-5e8c4493cd18\") "
Sep 16 14:54:41.904087 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:41.904055    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02a848a-daaa-419b-a0e1-5e8c4493cd18-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c02a848a-daaa-419b-a0e1-5e8c4493cd18" (UID: "c02a848a-daaa-419b-a0e1-5e8c4493cd18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:54:41.905745 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:41.905714    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02a848a-daaa-419b-a0e1-5e8c4493cd18-kube-api-access-fql52" (OuterVolumeSpecName: "kube-api-access-fql52") pod "c02a848a-daaa-419b-a0e1-5e8c4493cd18" (UID: "c02a848a-daaa-419b-a0e1-5e8c4493cd18"). InnerVolumeSpecName "kube-api-access-fql52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:54:41.991775 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:41.991740    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"]
Sep 16 14:54:42.001030 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.000995    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a743146e-8wqq6"]
Sep 16 14:54:42.004838 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.004810    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fql52\" (UniqueName: \"kubernetes.io/projected/c02a848a-daaa-419b-a0e1-5e8c4493cd18-kube-api-access-fql52\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:54:42.004958 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.004843    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c02a848a-daaa-419b-a0e1-5e8c4493cd18-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:54:42.403058 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.403031    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"]
Sep 16 14:54:42.403454 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.403442    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c02a848a-daaa-419b-a0e1-5e8c4493cd18" containerName="payload-processing"
Sep 16 14:54:42.403503 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.403456    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c02a848a-daaa-419b-a0e1-5e8c4493cd18" containerName="payload-processing"
Sep 16 14:54:42.403503 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.403473    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c6df49-fee3-4d12-a7c0-024ed86862fd" containerName="revoke-keys"
Sep 16 14:54:42.403503 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.403479    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c6df49-fee3-4d12-a7c0-024ed86862fd" containerName="revoke-keys"
Sep 16 14:54:42.403597 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.403535    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c6df49-fee3-4d12-a7c0-024ed86862fd" containerName="revoke-keys"
Sep 16 14:54:42.403597 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.403543    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c02a848a-daaa-419b-a0e1-5e8c4493cd18" containerName="payload-processing"
Sep 16 14:54:42.406685 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.406664    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"
Sep 16 14:54:42.409934 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.409915    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 14:54:42.426452 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.426426    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"]
Sep 16 14:54:42.507663 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.507624    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ae12b5-aebd-4a35-bdd2-93af2361a6d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-230a34-ldg6c\" (UID: \"60ae12b5-aebd-4a35-bdd2-93af2361a6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"
Sep 16 14:54:42.608725 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.608693    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ae12b5-aebd-4a35-bdd2-93af2361a6d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-230a34-ldg6c\" (UID: \"60ae12b5-aebd-4a35-bdd2-93af2361a6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"
Sep 16 14:54:42.609337 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.609318    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ae12b5-aebd-4a35-bdd2-93af2361a6d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-230a34-ldg6c\" (UID: \"60ae12b5-aebd-4a35-bdd2-93af2361a6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"
Sep 16 14:54:42.717161 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.717082    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"
Sep 16 14:54:42.793143 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.793105    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2" event={"ID":"c02a848a-daaa-419b-a0e1-5e8c4493cd18","Type":"ContainerDied","Data":"7e2bd2d8abb1651b25ed72e47b44cbdafb5c422ee6e23627d6784cf1982b3006"}
Sep 16 14:54:42.793143 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.793132    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"
Sep 16 14:54:42.793390 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.793204    2558 scope.go:117] "RemoveContainer" containerID="a9174cd978b8f355a19a9efdd8332308dcf54dfce3275e7eb18ebac1a968cd66"
Sep 16 14:54:42.819592 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.819492    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"]
Sep 16 14:54:42.822890 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.822862    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a743146e-df6bb76f-pswb2"]
Sep 16 14:54:42.991514 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:42.991492    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"]
Sep 16 14:54:42.993463 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:54:42.993432    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ae12b5_aebd_4a35_bdd2_93af2361a6d8.slice/crio-e4581c2b12799ab106520c291b86f5c252ffd68ff0293b18ca7b1eeb87b08813 WatchSource:0}: Error finding container e4581c2b12799ab106520c291b86f5c252ffd68ff0293b18ca7b1eeb87b08813: Status 404 returned error can't find the container with id e4581c2b12799ab106520c291b86f5c252ffd68ff0293b18ca7b1eeb87b08813
Sep 16 14:54:43.295653 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:43.295624    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c02a848a-daaa-419b-a0e1-5e8c4493cd18" path="/var/lib/kubelet/pods/c02a848a-daaa-419b-a0e1-5e8c4493cd18/volumes"
Sep 16 14:54:43.296048 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:43.296034    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1c6df49-fee3-4d12-a7c0-024ed86862fd" path="/var/lib/kubelet/pods/f1c6df49-fee3-4d12-a7c0-024ed86862fd/volumes"
Sep 16 14:54:43.801393 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:43.801365    2558 generic.go:356] "Generic (PLEG): container finished" podID="60ae12b5-aebd-4a35-bdd2-93af2361a6d8" containerID="b855cc629cabebdd962e71dd0efaeb06b09b54467a1d39f46c7b5f446583acca" exitCode=0
Sep 16 14:54:43.801781 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:43.801442    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c" event={"ID":"60ae12b5-aebd-4a35-bdd2-93af2361a6d8","Type":"ContainerDied","Data":"b855cc629cabebdd962e71dd0efaeb06b09b54467a1d39f46c7b5f446583acca"}
Sep 16 14:54:43.801781 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:43.801518    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c" event={"ID":"60ae12b5-aebd-4a35-bdd2-93af2361a6d8","Type":"ContainerStarted","Data":"e4581c2b12799ab106520c291b86f5c252ffd68ff0293b18ca7b1eeb87b08813"}
Sep 16 14:54:44.955137 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:44.955111    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"
Sep 16 14:54:45.033483 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:45.033450    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ae12b5-aebd-4a35-bdd2-93af2361a6d8-maas-api-service-ca\") pod \"60ae12b5-aebd-4a35-bdd2-93af2361a6d8\" (UID: \"60ae12b5-aebd-4a35-bdd2-93af2361a6d8\") "
Sep 16 14:54:45.033755 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:45.033735    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ae12b5-aebd-4a35-bdd2-93af2361a6d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60ae12b5-aebd-4a35-bdd2-93af2361a6d8" (UID: "60ae12b5-aebd-4a35-bdd2-93af2361a6d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:54:45.135255 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:45.135160    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60ae12b5-aebd-4a35-bdd2-93af2361a6d8-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:54:45.813921 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:45.813890    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"
Sep 16 14:54:45.814136 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:45.813924    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c" event={"ID":"60ae12b5-aebd-4a35-bdd2-93af2361a6d8","Type":"ContainerDied","Data":"e4581c2b12799ab106520c291b86f5c252ffd68ff0293b18ca7b1eeb87b08813"}
Sep 16 14:54:45.814136 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:45.813980    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4581c2b12799ab106520c291b86f5c252ffd68ff0293b18ca7b1eeb87b08813"
Sep 16 14:54:48.139887 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.139857    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"]
Sep 16 14:54:48.140283 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.140262    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ae12b5-aebd-4a35-bdd2-93af2361a6d8" containerName="revoke-keys"
Sep 16 14:54:48.140283 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.140274    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ae12b5-aebd-4a35-bdd2-93af2361a6d8" containerName="revoke-keys"
Sep 16 14:54:48.140379 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.140368    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="60ae12b5-aebd-4a35-bdd2-93af2361a6d8" containerName="revoke-keys"
Sep 16 14:54:48.143590 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.143565    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.147307 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.147285    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e33b3f\"/\"e2e-conflict-a-b025547b-kserve-self-signed-certs\""
Sep 16 14:54:48.160904 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.160876    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"]
Sep 16 14:54:48.164207 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.164181    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-model-cache\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.164334 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.164277    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tmp-dir\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.164334 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.164309    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tls-certs\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.164451 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.164404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hv5j\" (UniqueName: \"kubernetes.io/projected/a234d648-d7f9-4d31-aca6-6ce8c22384a7-kube-api-access-7hv5j\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.164507 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.164493    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-home\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.164582 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.164566    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-dshm\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.265641 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.265604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tmp-dir\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.265641 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.265643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tls-certs\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.265885 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.265687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hv5j\" (UniqueName: \"kubernetes.io/projected/a234d648-d7f9-4d31-aca6-6ce8c22384a7-kube-api-access-7hv5j\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.265885 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.265730    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-home\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.265885 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.265778    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-dshm\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.265885 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.265808    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-model-cache\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.266094 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.266049    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tmp-dir\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.266154 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.266104    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-home\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.266154 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.266131    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-model-cache\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.268287 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.268264    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-dshm\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.268599 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.268577    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tls-certs\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.275707 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.275679    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hv5j\" (UniqueName: \"kubernetes.io/projected/a234d648-d7f9-4d31-aca6-6ce8c22384a7-kube-api-access-7hv5j\") pod \"e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.452449 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.452362    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:48.727543 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.727517    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"]
Sep 16 14:54:48.729431 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:54:48.729400    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda234d648_d7f9_4d31_aca6_6ce8c22384a7.slice/crio-9902f63c0af23b4cda8ad26198ebad7c5240ffbd5228d13a08c87315de8da2bd WatchSource:0}: Error finding container 9902f63c0af23b4cda8ad26198ebad7c5240ffbd5228d13a08c87315de8da2bd: Status 404 returned error can't find the container with id 9902f63c0af23b4cda8ad26198ebad7c5240ffbd5228d13a08c87315de8da2bd
Sep 16 14:54:48.835014 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.834978    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5" event={"ID":"a234d648-d7f9-4d31-aca6-6ce8c22384a7","Type":"ContainerStarted","Data":"78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290"}
Sep 16 14:54:48.835014 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.835026    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5" event={"ID":"a234d648-d7f9-4d31-aca6-6ce8c22384a7","Type":"ContainerStarted","Data":"9902f63c0af23b4cda8ad26198ebad7c5240ffbd5228d13a08c87315de8da2bd"}
Sep 16 14:54:48.860742 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:48.860691    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5" podStartSLOduration=0.860671236 podStartE2EDuration="860.671236ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:54:48.859660735 +0000 UTC m=+1166.013739033" watchObservedRunningTime="2026-09-16 14:54:48.860671236 +0000 UTC m=+1166.014749523"
Sep 16 14:54:51.094619 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:51.094584    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"]
Sep 16 14:54:53.523139 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:53.523095    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"]
Sep 16 14:54:53.525035 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:53.525014    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-230a34-ldg6c"]
Sep 16 14:54:55.296251 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:55.296217    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60ae12b5-aebd-4a35-bdd2-93af2361a6d8" path="/var/lib/kubelet/pods/60ae12b5-aebd-4a35-bdd2-93af2361a6d8/volumes"
Sep 16 14:54:58.452818 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:58.452787    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:58.453258 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:58.452869    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:58.465411 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:58.465385    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:54:58.902943 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:54:58.902909    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:55:06.113541 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.113471    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt" podUID="78b53180-f8c7-46fa-b488-c19795c3374c" containerName="main" containerID="cri-o://85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2" gracePeriod=45
Sep 16 14:55:06.367317 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.367252    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:55:06.452023 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.451989    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-tmp-dir\") pod \"78b53180-f8c7-46fa-b488-c19795c3374c\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") "
Sep 16 14:55:06.452023 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.452026    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gdbj\" (UniqueName: \"kubernetes.io/projected/78b53180-f8c7-46fa-b488-c19795c3374c-kube-api-access-5gdbj\") pod \"78b53180-f8c7-46fa-b488-c19795c3374c\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") "
Sep 16 14:55:06.452320 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.452067    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b53180-f8c7-46fa-b488-c19795c3374c-tls-certs\") pod \"78b53180-f8c7-46fa-b488-c19795c3374c\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") "
Sep 16 14:55:06.452320 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.452242    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-model-cache\") pod \"78b53180-f8c7-46fa-b488-c19795c3374c\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") "
Sep 16 14:55:06.452320 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.452297    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-dshm\") pod \"78b53180-f8c7-46fa-b488-c19795c3374c\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") "
Sep 16 14:55:06.452493 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.452321    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78b53180-f8c7-46fa-b488-c19795c3374c" (UID: "78b53180-f8c7-46fa-b488-c19795c3374c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:06.452493 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.452408    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-home\") pod \"78b53180-f8c7-46fa-b488-c19795c3374c\" (UID: \"78b53180-f8c7-46fa-b488-c19795c3374c\") "
Sep 16 14:55:06.452608 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.452511    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-model-cache" (OuterVolumeSpecName: "model-cache") pod "78b53180-f8c7-46fa-b488-c19795c3374c" (UID: "78b53180-f8c7-46fa-b488-c19795c3374c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:06.452854 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.452799    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:55:06.452854 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.452825    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:55:06.453014 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.452881    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-home" (OuterVolumeSpecName: "home") pod "78b53180-f8c7-46fa-b488-c19795c3374c" (UID: "78b53180-f8c7-46fa-b488-c19795c3374c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:06.454499 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.454478    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78b53180-f8c7-46fa-b488-c19795c3374c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78b53180-f8c7-46fa-b488-c19795c3374c" (UID: "78b53180-f8c7-46fa-b488-c19795c3374c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:55:06.454622 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.454605    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-dshm" (OuterVolumeSpecName: "dshm") pod "78b53180-f8c7-46fa-b488-c19795c3374c" (UID: "78b53180-f8c7-46fa-b488-c19795c3374c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:55:06.454679 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.454605    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b53180-f8c7-46fa-b488-c19795c3374c-kube-api-access-5gdbj" (OuterVolumeSpecName: "kube-api-access-5gdbj") pod "78b53180-f8c7-46fa-b488-c19795c3374c" (UID: "78b53180-f8c7-46fa-b488-c19795c3374c"). InnerVolumeSpecName "kube-api-access-5gdbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:55:06.554287 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.554256    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:55:06.554287 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.554286    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gdbj\" (UniqueName: \"kubernetes.io/projected/78b53180-f8c7-46fa-b488-c19795c3374c-kube-api-access-5gdbj\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:55:06.554287 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.554296    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78b53180-f8c7-46fa-b488-c19795c3374c-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:55:06.554517 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.554304    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78b53180-f8c7-46fa-b488-c19795c3374c-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:55:06.939217 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.939160    2558 generic.go:356] "Generic (PLEG): container finished" podID="78b53180-f8c7-46fa-b488-c19795c3374c" containerID="85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2" exitCode=0
Sep 16 14:55:06.939217 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.939221    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"
Sep 16 14:55:06.939466 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.939225    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt" event={"ID":"78b53180-f8c7-46fa-b488-c19795c3374c","Type":"ContainerDied","Data":"85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2"}
Sep 16 14:55:06.939466 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.939263    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt" event={"ID":"78b53180-f8c7-46fa-b488-c19795c3374c","Type":"ContainerDied","Data":"71c5d814f3a135c19b978c17fdc260e23166b0bfdc87fabb1e13462f732db9dd"}
Sep 16 14:55:06.939466 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.939285    2558 scope.go:117] "RemoveContainer" containerID="85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2"
Sep 16 14:55:06.948975 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.948957    2558 scope.go:117] "RemoveContainer" containerID="85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2"
Sep 16 14:55:06.949257 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:55:06.949236    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2\": container with ID starting with 85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2 not found: ID does not exist" containerID="85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2"
Sep 16 14:55:06.949348 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.949264    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2"} err="failed to get container status \"85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2\": rpc error: code = NotFound desc = could not find container \"85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2\": container with ID starting with 85a718e6b90b0fec3ab905cea824d80195a60135031418e0eb7f7678bafcbbc2 not found: ID does not exist"
Sep 16 14:55:06.960923 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.960901    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"]
Sep 16 14:55:06.970131 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:06.970111    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-230a34/explicit-model-230a34-kserve-779794d67f-blvbt"]
Sep 16 14:55:07.295240 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:07.295146    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78b53180-f8c7-46fa-b488-c19795c3374c" path="/var/lib/kubelet/pods/78b53180-f8c7-46fa-b488-c19795c3374c/volumes"
Sep 16 14:55:23.261103 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:23.261074    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:55:23.264066 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:55:23.264046    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 14:56:08.614249 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:08.614216    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"]
Sep 16 14:56:08.614657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:08.614635    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b53180-f8c7-46fa-b488-c19795c3374c" containerName="main"
Sep 16 14:56:08.614657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:08.614648    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b53180-f8c7-46fa-b488-c19795c3374c" containerName="main"
Sep 16 14:56:08.614742 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:08.614729    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b53180-f8c7-46fa-b488-c19795c3374c" containerName="main"
Sep 16 14:56:08.617791 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:08.617775    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"
Sep 16 14:56:08.621394 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:08.621377    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 14:56:08.640305 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:08.640283    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"]
Sep 16 14:56:08.672842 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:08.672812    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dee1f3-37e8-4d6a-9e77-8f699a033385-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn\" (UID: \"87dee1f3-37e8-4d6a-9e77-8f699a033385\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"
Sep 16 14:56:08.773998 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:08.773962    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dee1f3-37e8-4d6a-9e77-8f699a033385-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn\" (UID: \"87dee1f3-37e8-4d6a-9e77-8f699a033385\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"
Sep 16 14:56:08.774637 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:08.774615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dee1f3-37e8-4d6a-9e77-8f699a033385-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn\" (UID: \"87dee1f3-37e8-4d6a-9e77-8f699a033385\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"
Sep 16 14:56:08.928120 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:08.928034    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"
Sep 16 14:56:09.225486 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:09.225456    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"]
Sep 16 14:56:09.225926 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:56:09.225901    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dee1f3_37e8_4d6a_9e77_8f699a033385.slice/crio-3b8f99517d6f9ba9cb1ca509062dc14eb059794cc8433c78f7ed4c316ba7d67a WatchSource:0}: Error finding container 3b8f99517d6f9ba9cb1ca509062dc14eb059794cc8433c78f7ed4c316ba7d67a: Status 404 returned error can't find the container with id 3b8f99517d6f9ba9cb1ca509062dc14eb059794cc8433c78f7ed4c316ba7d67a
Sep 16 14:56:09.282599 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:09.282572    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn" event={"ID":"87dee1f3-37e8-4d6a-9e77-8f699a033385","Type":"ContainerStarted","Data":"3b8f99517d6f9ba9cb1ca509062dc14eb059794cc8433c78f7ed4c316ba7d67a"}
Sep 16 14:56:10.290072 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:10.289995    2558 generic.go:356] "Generic (PLEG): container finished" podID="87dee1f3-37e8-4d6a-9e77-8f699a033385" containerID="8d2e90798b5aa54ec7629fa0f5fc98707296ca765a99c0f87b407c13aadc0895" exitCode=0
Sep 16 14:56:10.290072 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:10.290049    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn" event={"ID":"87dee1f3-37e8-4d6a-9e77-8f699a033385","Type":"ContainerDied","Data":"8d2e90798b5aa54ec7629fa0f5fc98707296ca765a99c0f87b407c13aadc0895"}
Sep 16 14:56:11.429723 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:11.429696    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"
Sep 16 14:56:11.501732 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:11.501701    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dee1f3-37e8-4d6a-9e77-8f699a033385-maas-api-service-ca\") pod \"87dee1f3-37e8-4d6a-9e77-8f699a033385\" (UID: \"87dee1f3-37e8-4d6a-9e77-8f699a033385\") "
Sep 16 14:56:11.502037 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:11.502017    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87dee1f3-37e8-4d6a-9e77-8f699a033385-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87dee1f3-37e8-4d6a-9e77-8f699a033385" (UID: "87dee1f3-37e8-4d6a-9e77-8f699a033385"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:56:11.603308 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:11.603275    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87dee1f3-37e8-4d6a-9e77-8f699a033385-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:56:12.302362 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:12.302332    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"
Sep 16 14:56:12.302536 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:12.302368    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn" event={"ID":"87dee1f3-37e8-4d6a-9e77-8f699a033385","Type":"ContainerDied","Data":"3b8f99517d6f9ba9cb1ca509062dc14eb059794cc8433c78f7ed4c316ba7d67a"}
Sep 16 14:56:12.302536 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:12.302411    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b8f99517d6f9ba9cb1ca509062dc14eb059794cc8433c78f7ed4c316ba7d67a"
Sep 16 14:56:23.348160 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:23.348076    2558 scope.go:117] "RemoveContainer" containerID="57ea95c814c79d06faaf00086d0bc3d9925a96e0490d47c68a5f63d43f47cd66"
Sep 16 14:56:26.119753 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.119715    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"]
Sep 16 14:56:26.120148 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.119933    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst" podUID="25842f7c-d1d7-4c83-9dbe-c3da0a4cffea" containerName="payload-pre-processing" containerID="cri-o://c8ab409c9bc9c607b5bd01ef3371b2758f8c70fb6e8bde4174890a1d67d730eb" gracePeriod=30
Sep 16 14:56:26.135056 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.135026    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"]
Sep 16 14:56:26.135324 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.135299    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67" podUID="78951ca2-2329-4c12-b181-5f4543fe4fef" containerName="payload-processing" containerID="cri-o://5c22c0c5826be72d0387a7c6f03c99f4f40881657443748e1847eb3463e0fdb1" gracePeriod=30
Sep 16 14:56:26.381158 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.381069    2558 generic.go:356] "Generic (PLEG): container finished" podID="25842f7c-d1d7-4c83-9dbe-c3da0a4cffea" containerID="c8ab409c9bc9c607b5bd01ef3371b2758f8c70fb6e8bde4174890a1d67d730eb" exitCode=0
Sep 16 14:56:26.381341 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.381147    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst" event={"ID":"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea","Type":"ContainerDied","Data":"c8ab409c9bc9c607b5bd01ef3371b2758f8c70fb6e8bde4174890a1d67d730eb"}
Sep 16 14:56:26.382870 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.382841    2558 generic.go:356] "Generic (PLEG): container finished" podID="78951ca2-2329-4c12-b181-5f4543fe4fef" containerID="5c22c0c5826be72d0387a7c6f03c99f4f40881657443748e1847eb3463e0fdb1" exitCode=0
Sep 16 14:56:26.382990 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.382899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67" event={"ID":"78951ca2-2329-4c12-b181-5f4543fe4fef","Type":"ContainerDied","Data":"5c22c0c5826be72d0387a7c6f03c99f4f40881657443748e1847eb3463e0fdb1"}
Sep 16 14:56:26.414875 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.414834    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:56:26.418075 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.418053    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:56:26.550157 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.550116    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lddx2\" (UniqueName: \"kubernetes.io/projected/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-kube-api-access-lddx2\") pod \"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea\" (UID: \"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea\") "
Sep 16 14:56:26.550454 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.550226    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmp6n\" (UniqueName: \"kubernetes.io/projected/78951ca2-2329-4c12-b181-5f4543fe4fef-kube-api-access-xmp6n\") pod \"78951ca2-2329-4c12-b181-5f4543fe4fef\" (UID: \"78951ca2-2329-4c12-b181-5f4543fe4fef\") "
Sep 16 14:56:26.550454 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.550301    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-plugins-config-volume\") pod \"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea\" (UID: \"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea\") "
Sep 16 14:56:26.550454 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.550332    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78951ca2-2329-4c12-b181-5f4543fe4fef-plugins-config-volume\") pod \"78951ca2-2329-4c12-b181-5f4543fe4fef\" (UID: \"78951ca2-2329-4c12-b181-5f4543fe4fef\") "
Sep 16 14:56:26.550708 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.550678    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25842f7c-d1d7-4c83-9dbe-c3da0a4cffea" (UID: "25842f7c-d1d7-4c83-9dbe-c3da0a4cffea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:56:26.550803 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.550778    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78951ca2-2329-4c12-b181-5f4543fe4fef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78951ca2-2329-4c12-b181-5f4543fe4fef" (UID: "78951ca2-2329-4c12-b181-5f4543fe4fef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:56:26.551154 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.551131    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:56:26.551270 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.551157    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78951ca2-2329-4c12-b181-5f4543fe4fef-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:56:26.552536 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.552507    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-kube-api-access-lddx2" (OuterVolumeSpecName: "kube-api-access-lddx2") pod "25842f7c-d1d7-4c83-9dbe-c3da0a4cffea" (UID: "25842f7c-d1d7-4c83-9dbe-c3da0a4cffea"). InnerVolumeSpecName "kube-api-access-lddx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:56:26.552626 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.552584    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78951ca2-2329-4c12-b181-5f4543fe4fef-kube-api-access-xmp6n" (OuterVolumeSpecName: "kube-api-access-xmp6n") pod "78951ca2-2329-4c12-b181-5f4543fe4fef" (UID: "78951ca2-2329-4c12-b181-5f4543fe4fef"). InnerVolumeSpecName "kube-api-access-xmp6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:56:26.652706 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.652620    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lddx2\" (UniqueName: \"kubernetes.io/projected/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea-kube-api-access-lddx2\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:56:26.652706 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.652654    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmp6n\" (UniqueName: \"kubernetes.io/projected/78951ca2-2329-4c12-b181-5f4543fe4fef-kube-api-access-xmp6n\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:56:26.714721 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.714685    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"]
Sep 16 14:56:26.730611 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:26.730576    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-78e52a-j84rn"]
Sep 16 14:56:27.296265 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:27.296232    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87dee1f3-37e8-4d6a-9e77-8f699a033385" path="/var/lib/kubelet/pods/87dee1f3-37e8-4d6a-9e77-8f699a033385/volumes"
Sep 16 14:56:27.390203 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:27.390151    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"
Sep 16 14:56:27.390203 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:27.390196    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst" event={"ID":"25842f7c-d1d7-4c83-9dbe-c3da0a4cffea","Type":"ContainerDied","Data":"ed7e11234fae67f612679575f0532989c6356e939588180f8357ce0669ca8fc6"}
Sep 16 14:56:27.390472 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:27.390257    2558 scope.go:117] "RemoveContainer" containerID="c8ab409c9bc9c607b5bd01ef3371b2758f8c70fb6e8bde4174890a1d67d730eb"
Sep 16 14:56:27.391852 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:27.391777    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"
Sep 16 14:56:27.391852 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:27.391799    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67" event={"ID":"78951ca2-2329-4c12-b181-5f4543fe4fef","Type":"ContainerDied","Data":"f494c7aa9245f31986132d9f3d049414ab9933b1f3751bab9765a6be0d00e46f"}
Sep 16 14:56:27.399657 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:27.399635    2558 scope.go:117] "RemoveContainer" containerID="5c22c0c5826be72d0387a7c6f03c99f4f40881657443748e1847eb3463e0fdb1"
Sep 16 14:56:27.428497 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:27.428467    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"]
Sep 16 14:56:27.440561 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:27.440530    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-78e52a-556c64d874-vsdst"]
Sep 16 14:56:27.465375 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:27.465345    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"]
Sep 16 14:56:27.472282 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:27.472255    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-78e52a-86c977d8c7-k9m67"]
Sep 16 14:56:29.295659 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:29.295627    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25842f7c-d1d7-4c83-9dbe-c3da0a4cffea" path="/var/lib/kubelet/pods/25842f7c-d1d7-4c83-9dbe-c3da0a4cffea/volumes"
Sep 16 14:56:29.296043 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:29.295969    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78951ca2-2329-4c12-b181-5f4543fe4fef" path="/var/lib/kubelet/pods/78951ca2-2329-4c12-b181-5f4543fe4fef/volumes"
Sep 16 14:56:34.500632 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.500590    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"]
Sep 16 14:56:34.501204 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.501184    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25842f7c-d1d7-4c83-9dbe-c3da0a4cffea" containerName="payload-pre-processing"
Sep 16 14:56:34.501204 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.501205    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="25842f7c-d1d7-4c83-9dbe-c3da0a4cffea" containerName="payload-pre-processing"
Sep 16 14:56:34.501357 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.501224    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78951ca2-2329-4c12-b181-5f4543fe4fef" containerName="payload-processing"
Sep 16 14:56:34.501357 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.501232    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="78951ca2-2329-4c12-b181-5f4543fe4fef" containerName="payload-processing"
Sep 16 14:56:34.501357 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.501246    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87dee1f3-37e8-4d6a-9e77-8f699a033385" containerName="revoke-keys"
Sep 16 14:56:34.501357 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.501255    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="87dee1f3-37e8-4d6a-9e77-8f699a033385" containerName="revoke-keys"
Sep 16 14:56:34.501564 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.501371    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="25842f7c-d1d7-4c83-9dbe-c3da0a4cffea" containerName="payload-pre-processing"
Sep 16 14:56:34.501564 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.501387    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="87dee1f3-37e8-4d6a-9e77-8f699a033385" containerName="revoke-keys"
Sep 16 14:56:34.501564 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.501398    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="78951ca2-2329-4c12-b181-5f4543fe4fef" containerName="payload-processing"
Sep 16 14:56:34.505967 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.505946    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.508696 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:56:34.508663    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"noten-model-f4d89a-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-128-123.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-noten-f4d89a\": no relationship found between node 'ip-10-0-128-123.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-noten-f4d89a\"/\"noten-model-f4d89a-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 16 14:56:34.512066 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:56:34.512044    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-123.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-noten-f4d89a\": no relationship found between node 'ip-10-0-128-123.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-noten-f4d89a\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 16 14:56:34.514044 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:56:34.514012    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"noten-model-f4d89a-kserve-578d7f599-kdssr\" is forbidden: User \"system:node:ip-10-0-128-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-noten-f4d89a\": no relationship found between node 'ip-10-0-128-123.ec2.internal' and this object" podUID="046a91ef-c653-425e-8f6a-1cb9758c5817" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.519111 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:56:34.519088    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-123.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"ai-tenant-e2e-noten-f4d89a\": no relationship found between node 'ip-10-0-128-123.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-noten-f4d89a\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 14:56:34.519229 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:56:34.519183    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-xc988\" is forbidden: User \"system:node:ip-10-0-128-123.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-noten-f4d89a\": no relationship found between node 'ip-10-0-128-123.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-noten-f4d89a\"/\"default-dockercfg-xc988\"" type="*v1.Secret"
Sep 16 14:56:34.519543 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.519527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046a91ef-c653-425e-8f6a-1cb9758c5817-tls-certs\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.519605 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.519589    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-dshm\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.519649 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.519617    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6pq\" (UniqueName: \"kubernetes.io/projected/046a91ef-c653-425e-8f6a-1cb9758c5817-kube-api-access-rk6pq\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.519649 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.519640    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-home\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.519758 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.519737    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-tmp-dir\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.519810 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.519776    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-model-cache\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.568848 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.568822    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"]
Sep 16 14:56:34.620288 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.620258    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6pq\" (UniqueName: \"kubernetes.io/projected/046a91ef-c653-425e-8f6a-1cb9758c5817-kube-api-access-rk6pq\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.620288 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.620297    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-home\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.620540 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.620339    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-tmp-dir\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.620540 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.620357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-model-cache\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.620540 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.620406    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046a91ef-c653-425e-8f6a-1cb9758c5817-tls-certs\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.620540 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.620462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-dshm\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.620764 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.620743    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-tmp-dir\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.620829 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.620792    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-home\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.620871 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.620824    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-model-cache\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:34.622682 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:34.622659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-dshm\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:35.436017 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:35.435989    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f4d89a\"/\"openshift-service-ca.crt\""
Sep 16 14:56:35.621653 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:56:35.621622    2558 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition
Sep 16 14:56:35.621986 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:56:35.621693    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/046a91ef-c653-425e-8f6a-1cb9758c5817-tls-certs podName:046a91ef-c653-425e-8f6a-1cb9758c5817 nodeName:}" failed. No retries permitted until 2026-09-16 14:56:36.121677108 +0000 UTC m=+1273.275755372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/046a91ef-c653-425e-8f6a-1cb9758c5817-tls-certs") pod "noten-model-f4d89a-kserve-578d7f599-kdssr" (UID: "046a91ef-c653-425e-8f6a-1cb9758c5817") : failed to sync secret cache: timed out waiting for the condition
Sep 16 14:56:35.624341 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:35.624323    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f4d89a\"/\"noten-model-f4d89a-kserve-self-signed-certs\""
Sep 16 14:56:35.638446 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:56:35.638426    2558 projected.go:291] Couldn't get configMap ai-tenant-e2e-noten-f4d89a/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 16 14:56:35.638558 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:56:35.638454    2558 projected.go:196] Error preparing data for projected volume kube-api-access-rk6pq for pod ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr: failed to sync configmap cache: timed out waiting for the condition
Sep 16 14:56:35.638558 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:56:35.638514    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/046a91ef-c653-425e-8f6a-1cb9758c5817-kube-api-access-rk6pq podName:046a91ef-c653-425e-8f6a-1cb9758c5817 nodeName:}" failed. No retries permitted until 2026-09-16 14:56:36.138500975 +0000 UTC m=+1273.292579241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rk6pq" (UniqueName: "kubernetes.io/projected/046a91ef-c653-425e-8f6a-1cb9758c5817-kube-api-access-rk6pq") pod "noten-model-f4d89a-kserve-578d7f599-kdssr" (UID: "046a91ef-c653-425e-8f6a-1cb9758c5817") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 14:56:35.807788 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:35.807759    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-f4d89a\"/\"kube-root-ca.crt\""
Sep 16 14:56:35.820353 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:35.820335    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-f4d89a\"/\"default-dockercfg-xc988\""
Sep 16 14:56:36.134205 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:36.134115    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046a91ef-c653-425e-8f6a-1cb9758c5817-tls-certs\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:36.136597 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:36.136579    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046a91ef-c653-425e-8f6a-1cb9758c5817-tls-certs\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:36.235588 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:36.235559    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6pq\" (UniqueName: \"kubernetes.io/projected/046a91ef-c653-425e-8f6a-1cb9758c5817-kube-api-access-rk6pq\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:36.237989 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:36.237965    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6pq\" (UniqueName: \"kubernetes.io/projected/046a91ef-c653-425e-8f6a-1cb9758c5817-kube-api-access-rk6pq\") pod \"noten-model-f4d89a-kserve-578d7f599-kdssr\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") " pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:36.316450 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:36.316413    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:36.628294 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:36.628271    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"]
Sep 16 14:56:36.629821 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:56:36.629787    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod046a91ef_c653_425e_8f6a_1cb9758c5817.slice/crio-064e65bb32c5c3e09322f78a22719e94d117408c278557af4c6af8c3ace4ad37 WatchSource:0}: Error finding container 064e65bb32c5c3e09322f78a22719e94d117408c278557af4c6af8c3ace4ad37: Status 404 returned error can't find the container with id 064e65bb32c5c3e09322f78a22719e94d117408c278557af4c6af8c3ace4ad37
Sep 16 14:56:37.448752 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:37.448719    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr" event={"ID":"046a91ef-c653-425e-8f6a-1cb9758c5817","Type":"ContainerStarted","Data":"23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33"}
Sep 16 14:56:37.448752 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:37.448758    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr" event={"ID":"046a91ef-c653-425e-8f6a-1cb9758c5817","Type":"ContainerStarted","Data":"064e65bb32c5c3e09322f78a22719e94d117408c278557af4c6af8c3ace4ad37"}
Sep 16 14:56:37.487818 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:37.487773    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr" podStartSLOduration=3.487757712 podStartE2EDuration="3.487757712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:56:37.486523869 +0000 UTC m=+1274.640602157" watchObservedRunningTime="2026-09-16 14:56:37.487757712 +0000 UTC m=+1274.641835998"
Sep 16 14:56:38.895472 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:38.895439    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"]
Sep 16 14:56:38.898984 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:38.898967    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:56:38.906479 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:38.906459    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-adb97937\""
Sep 16 14:56:38.922796 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:38.922772    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"]
Sep 16 14:56:38.962340 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:38.962313    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ead3d07-f374-4f6b-bcbf-d32502009563-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd\" (UID: \"8ead3d07-f374-4f6b-bcbf-d32502009563\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:56:38.962488 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:38.962430    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbxl\" (UniqueName: \"kubernetes.io/projected/8ead3d07-f374-4f6b-bcbf-d32502009563-kube-api-access-phbxl\") pod \"payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd\" (UID: \"8ead3d07-f374-4f6b-bcbf-d32502009563\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:56:39.063369 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.063336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbxl\" (UniqueName: \"kubernetes.io/projected/8ead3d07-f374-4f6b-bcbf-d32502009563-kube-api-access-phbxl\") pod \"payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd\" (UID: \"8ead3d07-f374-4f6b-bcbf-d32502009563\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:56:39.063530 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.063401    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ead3d07-f374-4f6b-bcbf-d32502009563-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd\" (UID: \"8ead3d07-f374-4f6b-bcbf-d32502009563\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:56:39.064059 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.064040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ead3d07-f374-4f6b-bcbf-d32502009563-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd\" (UID: \"8ead3d07-f374-4f6b-bcbf-d32502009563\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:56:39.081146 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.081119    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbxl\" (UniqueName: \"kubernetes.io/projected/8ead3d07-f374-4f6b-bcbf-d32502009563-kube-api-access-phbxl\") pod \"payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd\" (UID: \"8ead3d07-f374-4f6b-bcbf-d32502009563\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:56:39.132618 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.132594    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"]
Sep 16 14:56:39.136295 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.136277    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:56:39.141262 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.141242    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-adb97937-dockercfg-lwd74\""
Sep 16 14:56:39.161952 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.161904    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"]
Sep 16 14:56:39.165370 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.163980    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czwfp\" (UniqueName: \"kubernetes.io/projected/1c897f19-ca8b-4262-b862-2bbeea724281-kube-api-access-czwfp\") pod \"payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9\" (UID: \"1c897f19-ca8b-4262-b862-2bbeea724281\") " pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:56:39.165370 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.164046    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c897f19-ca8b-4262-b862-2bbeea724281-plugins-config-volume\") pod \"payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9\" (UID: \"1c897f19-ca8b-4262-b862-2bbeea724281\") " pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:56:39.210708 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.210683    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:56:39.264742 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.264716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czwfp\" (UniqueName: \"kubernetes.io/projected/1c897f19-ca8b-4262-b862-2bbeea724281-kube-api-access-czwfp\") pod \"payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9\" (UID: \"1c897f19-ca8b-4262-b862-2bbeea724281\") " pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:56:39.264858 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.264791    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c897f19-ca8b-4262-b862-2bbeea724281-plugins-config-volume\") pod \"payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9\" (UID: \"1c897f19-ca8b-4262-b862-2bbeea724281\") " pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:56:39.265886 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.265837    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c897f19-ca8b-4262-b862-2bbeea724281-plugins-config-volume\") pod \"payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9\" (UID: \"1c897f19-ca8b-4262-b862-2bbeea724281\") " pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:56:39.285315 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.285284    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czwfp\" (UniqueName: \"kubernetes.io/projected/1c897f19-ca8b-4262-b862-2bbeea724281-kube-api-access-czwfp\") pod \"payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9\" (UID: \"1c897f19-ca8b-4262-b862-2bbeea724281\") " pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:56:39.446402 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.446318    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:56:39.533007 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.532975    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"]
Sep 16 14:56:39.535639 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:56:39.535599    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ead3d07_f374_4f6b_bcbf_d32502009563.slice/crio-7a9f74cb9f67311f071ee090b5aed454de3ada26af746a2e7dfb0d944acc1d8e WatchSource:0}: Error finding container 7a9f74cb9f67311f071ee090b5aed454de3ada26af746a2e7dfb0d944acc1d8e: Status 404 returned error can't find the container with id 7a9f74cb9f67311f071ee090b5aed454de3ada26af746a2e7dfb0d944acc1d8e
Sep 16 14:56:39.762287 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:39.762261    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"]
Sep 16 14:56:39.764200 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:56:39.764149    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c897f19_ca8b_4262_b862_2bbeea724281.slice/crio-02b3448bfed7dd3bd19ef1c62719240a232b4dbc87f815a89ea321be388293ca WatchSource:0}: Error finding container 02b3448bfed7dd3bd19ef1c62719240a232b4dbc87f815a89ea321be388293ca: Status 404 returned error can't find the container with id 02b3448bfed7dd3bd19ef1c62719240a232b4dbc87f815a89ea321be388293ca
Sep 16 14:56:40.008157 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.008127    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"]
Sep 16 14:56:40.011725 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.011678    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"
Sep 16 14:56:40.015465 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.015444    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 14:56:40.039369 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.039342    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"]
Sep 16 14:56:40.071978 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.071946    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9feed5-2cc2-48d9-ab7b-67c94b28843b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp\" (UID: \"1c9feed5-2cc2-48d9-ab7b-67c94b28843b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"
Sep 16 14:56:40.172622 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.172592    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9feed5-2cc2-48d9-ab7b-67c94b28843b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp\" (UID: \"1c9feed5-2cc2-48d9-ab7b-67c94b28843b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"
Sep 16 14:56:40.173158 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.173141    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9feed5-2cc2-48d9-ab7b-67c94b28843b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp\" (UID: \"1c9feed5-2cc2-48d9-ab7b-67c94b28843b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"
Sep 16 14:56:40.321869 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.321842    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"
Sep 16 14:56:40.471188 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.469605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd" event={"ID":"8ead3d07-f374-4f6b-bcbf-d32502009563","Type":"ContainerStarted","Data":"0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d"}
Sep 16 14:56:40.471188 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.469667    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd" event={"ID":"8ead3d07-f374-4f6b-bcbf-d32502009563","Type":"ContainerStarted","Data":"7a9f74cb9f67311f071ee090b5aed454de3ada26af746a2e7dfb0d944acc1d8e"}
Sep 16 14:56:40.471188 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.469709    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:56:40.471492 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.471477    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9" event={"ID":"1c897f19-ca8b-4262-b862-2bbeea724281","Type":"ContainerStarted","Data":"393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b"}
Sep 16 14:56:40.471551 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.471520    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9" event={"ID":"1c897f19-ca8b-4262-b862-2bbeea724281","Type":"ContainerStarted","Data":"02b3448bfed7dd3bd19ef1c62719240a232b4dbc87f815a89ea321be388293ca"}
Sep 16 14:56:40.472373 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.471939    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:56:40.515058 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.515003    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd" podStartSLOduration=2.5149815760000003 podStartE2EDuration="2.514981576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:56:40.512992372 +0000 UTC m=+1277.667070661" watchObservedRunningTime="2026-09-16 14:56:40.514981576 +0000 UTC m=+1277.669059866"
Sep 16 14:56:40.545278 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.545234    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9" podStartSLOduration=1.545219555 podStartE2EDuration="1.545219555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:56:40.544079074 +0000 UTC m=+1277.698157360" watchObservedRunningTime="2026-09-16 14:56:40.545219555 +0000 UTC m=+1277.699297842"
Sep 16 14:56:40.635673 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:40.635644    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"]
Sep 16 14:56:40.636116 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:56:40.636085    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9feed5_2cc2_48d9_ab7b_67c94b28843b.slice/crio-4378d4afae20ab0c9990de30d51ce2711344c5cca4dad51795858c23a702f749 WatchSource:0}: Error finding container 4378d4afae20ab0c9990de30d51ce2711344c5cca4dad51795858c23a702f749: Status 404 returned error can't find the container with id 4378d4afae20ab0c9990de30d51ce2711344c5cca4dad51795858c23a702f749
Sep 16 14:56:41.480126 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:41.480091    2558 generic.go:356] "Generic (PLEG): container finished" podID="1c9feed5-2cc2-48d9-ab7b-67c94b28843b" containerID="9b060c70dc1d12cf42a02f7c21c1adbf2e0d2d948da144424c14320680d6d084" exitCode=0
Sep 16 14:56:41.480604 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:41.480185    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp" event={"ID":"1c9feed5-2cc2-48d9-ab7b-67c94b28843b","Type":"ContainerDied","Data":"9b060c70dc1d12cf42a02f7c21c1adbf2e0d2d948da144424c14320680d6d084"}
Sep 16 14:56:41.480604 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:41.480238    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp" event={"ID":"1c9feed5-2cc2-48d9-ab7b-67c94b28843b","Type":"ContainerStarted","Data":"4378d4afae20ab0c9990de30d51ce2711344c5cca4dad51795858c23a702f749"}
Sep 16 14:56:42.629645 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:42.629621    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"
Sep 16 14:56:42.698409 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:42.698382    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9feed5-2cc2-48d9-ab7b-67c94b28843b-maas-api-service-ca\") pod \"1c9feed5-2cc2-48d9-ab7b-67c94b28843b\" (UID: \"1c9feed5-2cc2-48d9-ab7b-67c94b28843b\") "
Sep 16 14:56:42.698747 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:42.698726    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9feed5-2cc2-48d9-ab7b-67c94b28843b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c9feed5-2cc2-48d9-ab7b-67c94b28843b" (UID: "1c9feed5-2cc2-48d9-ab7b-67c94b28843b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:56:42.799713 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:42.799631    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c9feed5-2cc2-48d9-ab7b-67c94b28843b-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:56:43.493633 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:43.493605    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"
Sep 16 14:56:43.493785 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:43.493607    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp" event={"ID":"1c9feed5-2cc2-48d9-ab7b-67c94b28843b","Type":"ContainerDied","Data":"4378d4afae20ab0c9990de30d51ce2711344c5cca4dad51795858c23a702f749"}
Sep 16 14:56:43.493785 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:43.493711    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4378d4afae20ab0c9990de30d51ce2711344c5cca4dad51795858c23a702f749"
Sep 16 14:56:46.317255 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:46.317223    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:46.317255 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:46.317262    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:46.329689 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:46.329660    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:46.522947 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:46.522919    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:56:50.398286 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.398257    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"]
Sep 16 14:56:50.398654 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.398450    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n" podUID="4ac36623-d4f5-4b61-add2-6abf1b55118e" containerName="payload-processing" containerID="cri-o://5c71afd255b8aee652fec1587d22476dc710757813fc9bafeb66c3c129abdc7e" gracePeriod=30
Sep 16 14:56:50.419547 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.419521    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"]
Sep 16 14:56:50.419763 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.419717    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7" podUID="a8053eff-d86e-41f2-a928-b347d12f5034" containerName="payload-pre-processing" containerID="cri-o://f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2" gracePeriod=30
Sep 16 14:56:50.535507 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.535409    2558 generic.go:356] "Generic (PLEG): container finished" podID="4ac36623-d4f5-4b61-add2-6abf1b55118e" containerID="5c71afd255b8aee652fec1587d22476dc710757813fc9bafeb66c3c129abdc7e" exitCode=0
Sep 16 14:56:50.535645 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.535515    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n" event={"ID":"4ac36623-d4f5-4b61-add2-6abf1b55118e","Type":"ContainerDied","Data":"5c71afd255b8aee652fec1587d22476dc710757813fc9bafeb66c3c129abdc7e"}
Sep 16 14:56:50.710723 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.710701    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:56:50.713513 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.713495    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:56:50.777942 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.777913    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4znbh\" (UniqueName: \"kubernetes.io/projected/a8053eff-d86e-41f2-a928-b347d12f5034-kube-api-access-4znbh\") pod \"a8053eff-d86e-41f2-a928-b347d12f5034\" (UID: \"a8053eff-d86e-41f2-a928-b347d12f5034\") "
Sep 16 14:56:50.778087 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.778012    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8053eff-d86e-41f2-a928-b347d12f5034-plugins-config-volume\") pod \"a8053eff-d86e-41f2-a928-b347d12f5034\" (UID: \"a8053eff-d86e-41f2-a928-b347d12f5034\") "
Sep 16 14:56:50.778134 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.778103    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac36623-d4f5-4b61-add2-6abf1b55118e-plugins-config-volume\") pod \"4ac36623-d4f5-4b61-add2-6abf1b55118e\" (UID: \"4ac36623-d4f5-4b61-add2-6abf1b55118e\") "
Sep 16 14:56:50.778204 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.778153    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcs92\" (UniqueName: \"kubernetes.io/projected/4ac36623-d4f5-4b61-add2-6abf1b55118e-kube-api-access-hcs92\") pod \"4ac36623-d4f5-4b61-add2-6abf1b55118e\" (UID: \"4ac36623-d4f5-4b61-add2-6abf1b55118e\") "
Sep 16 14:56:50.778479 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.778449    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8053eff-d86e-41f2-a928-b347d12f5034-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8053eff-d86e-41f2-a928-b347d12f5034" (UID: "a8053eff-d86e-41f2-a928-b347d12f5034"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:56:50.778582 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.778508    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac36623-d4f5-4b61-add2-6abf1b55118e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ac36623-d4f5-4b61-add2-6abf1b55118e" (UID: "4ac36623-d4f5-4b61-add2-6abf1b55118e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:56:50.780107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.780082    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8053eff-d86e-41f2-a928-b347d12f5034-kube-api-access-4znbh" (OuterVolumeSpecName: "kube-api-access-4znbh") pod "a8053eff-d86e-41f2-a928-b347d12f5034" (UID: "a8053eff-d86e-41f2-a928-b347d12f5034"). InnerVolumeSpecName "kube-api-access-4znbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:56:50.780282 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.780262    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac36623-d4f5-4b61-add2-6abf1b55118e-kube-api-access-hcs92" (OuterVolumeSpecName: "kube-api-access-hcs92") pod "4ac36623-d4f5-4b61-add2-6abf1b55118e" (UID: "4ac36623-d4f5-4b61-add2-6abf1b55118e"). InnerVolumeSpecName "kube-api-access-hcs92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:56:50.879860 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.879821    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8053eff-d86e-41f2-a928-b347d12f5034-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:56:50.879860 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.879854    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac36623-d4f5-4b61-add2-6abf1b55118e-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:56:50.879860 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.879866    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcs92\" (UniqueName: \"kubernetes.io/projected/4ac36623-d4f5-4b61-add2-6abf1b55118e-kube-api-access-hcs92\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:56:50.880090 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.879878    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4znbh\" (UniqueName: \"kubernetes.io/projected/a8053eff-d86e-41f2-a928-b347d12f5034-kube-api-access-4znbh\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:56:50.890050 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.890012    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"]
Sep 16 14:56:50.901977 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:50.901951    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e689e5-2b7kp"]
Sep 16 14:56:51.296927 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.296889    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c9feed5-2cc2-48d9-ab7b-67c94b28843b" path="/var/lib/kubelet/pods/1c9feed5-2cc2-48d9-ab7b-67c94b28843b/volumes"
Sep 16 14:56:51.483016 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.482989    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:56:51.483460 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.483204    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:56:51.542531 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.542495    2558 generic.go:356] "Generic (PLEG): container finished" podID="a8053eff-d86e-41f2-a928-b347d12f5034" containerID="f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2" exitCode=0
Sep 16 14:56:51.542730 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.542538    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"
Sep 16 14:56:51.542730 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.542582    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7" event={"ID":"a8053eff-d86e-41f2-a928-b347d12f5034","Type":"ContainerDied","Data":"f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2"}
Sep 16 14:56:51.542730 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.542637    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7" event={"ID":"a8053eff-d86e-41f2-a928-b347d12f5034","Type":"ContainerDied","Data":"3acf1079cf92efeb972837d2b5630c50e2b3f6de47bb7b4d3317fe780bd1327a"}
Sep 16 14:56:51.542730 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.542665    2558 scope.go:117] "RemoveContainer" containerID="f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2"
Sep 16 14:56:51.543989 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.543969    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n" event={"ID":"4ac36623-d4f5-4b61-add2-6abf1b55118e","Type":"ContainerDied","Data":"e9094ef78dcb4f2a01dad0ea2dded2080e26cc3b58590806938fecb7d8851df9"}
Sep 16 14:56:51.544080 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.544010    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"
Sep 16 14:56:51.551402 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.551378    2558 scope.go:117] "RemoveContainer" containerID="f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2"
Sep 16 14:56:51.551672 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:56:51.551654    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2\": container with ID starting with f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2 not found: ID does not exist" containerID="f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2"
Sep 16 14:56:51.551737 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.551680    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2"} err="failed to get container status \"f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2\": rpc error: code = NotFound desc = could not find container \"f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2\": container with ID starting with f871d03090fd36a4c00e3e080d0c63931ed81117236125f9b76873c4b6093bf2 not found: ID does not exist"
Sep 16 14:56:51.551737 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.551695    2558 scope.go:117] "RemoveContainer" containerID="5c71afd255b8aee652fec1587d22476dc710757813fc9bafeb66c3c129abdc7e"
Sep 16 14:56:51.621946 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.621904    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"]
Sep 16 14:56:51.630775 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.630745    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e689e5-669cdb9c66-4sdg7"]
Sep 16 14:56:51.716383 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.716346    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"]
Sep 16 14:56:51.726358 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:51.726330    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e689e5-5745678df9-5845n"]
Sep 16 14:56:53.296891 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:53.296860    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ac36623-d4f5-4b61-add2-6abf1b55118e" path="/var/lib/kubelet/pods/4ac36623-d4f5-4b61-add2-6abf1b55118e/volumes"
Sep 16 14:56:53.297419 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:53.297259    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8053eff-d86e-41f2-a928-b347d12f5034" path="/var/lib/kubelet/pods/a8053eff-d86e-41f2-a928-b347d12f5034/volumes"
Sep 16 14:56:57.910730 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.910699    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"]
Sep 16 14:56:57.911231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.911087    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c9feed5-2cc2-48d9-ab7b-67c94b28843b" containerName="revoke-keys"
Sep 16 14:56:57.911231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.911099    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c9feed5-2cc2-48d9-ab7b-67c94b28843b" containerName="revoke-keys"
Sep 16 14:56:57.911231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.911114    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8053eff-d86e-41f2-a928-b347d12f5034" containerName="payload-pre-processing"
Sep 16 14:56:57.911231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.911120    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8053eff-d86e-41f2-a928-b347d12f5034" containerName="payload-pre-processing"
Sep 16 14:56:57.911231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.911138    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac36623-d4f5-4b61-add2-6abf1b55118e" containerName="payload-processing"
Sep 16 14:56:57.911231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.911144    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac36623-d4f5-4b61-add2-6abf1b55118e" containerName="payload-processing"
Sep 16 14:56:57.911231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.911214    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8053eff-d86e-41f2-a928-b347d12f5034" containerName="payload-pre-processing"
Sep 16 14:56:57.911231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.911224    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac36623-d4f5-4b61-add2-6abf1b55118e" containerName="payload-processing"
Sep 16 14:56:57.911231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.911232    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c9feed5-2cc2-48d9-ab7b-67c94b28843b" containerName="revoke-keys"
Sep 16 14:56:57.914109 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.914089    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:57.917494 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.917471    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e33b3f\"/\"e2e-conflict-b-b025547b-kserve-self-signed-certs\""
Sep 16 14:56:57.945207 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:57.945183    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"]
Sep 16 14:56:58.054832 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.054803    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tls-certs\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.055009 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.054871    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-dshm\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.055009 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.054907    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-model-cache\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.055009 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.054937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-home\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.055138 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.055007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tmp-dir\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.055138 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.055034    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nfp\" (UniqueName: \"kubernetes.io/projected/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-kube-api-access-49nfp\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.155562 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.155530    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-model-cache\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.155729 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.155576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-home\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.155729 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.155607    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tmp-dir\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.155729 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.155622    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nfp\" (UniqueName: \"kubernetes.io/projected/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-kube-api-access-49nfp\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.155893 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.155753    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tls-certs\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.155951 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.155888    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-dshm\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.155998 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.155968    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-home\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.156035 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.155996    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-model-cache\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.156035 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.156020    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tmp-dir\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.158037 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.158005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-dshm\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.158367 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.158349    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tls-certs\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.169182 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.169113    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nfp\" (UniqueName: \"kubernetes.io/projected/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-kube-api-access-49nfp\") pod \"e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") " pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.223640 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.223617    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:56:58.528908 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.528831    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"]
Sep 16 14:56:58.531695 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:56:58.531669    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd97fa8f1_1be2_4088_8dd4_92fc90f18c5d.slice/crio-e77b5303ead89fe4e484975802071271599b4d7c75184f40c7be03d6ada8d791 WatchSource:0}: Error finding container e77b5303ead89fe4e484975802071271599b4d7c75184f40c7be03d6ada8d791: Status 404 returned error can't find the container with id e77b5303ead89fe4e484975802071271599b4d7c75184f40c7be03d6ada8d791
Sep 16 14:56:58.588880 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.588855    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q" event={"ID":"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d","Type":"ContainerStarted","Data":"39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089"}
Sep 16 14:56:58.588975 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.588895    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q" event={"ID":"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d","Type":"ContainerStarted","Data":"e77b5303ead89fe4e484975802071271599b4d7c75184f40c7be03d6ada8d791"}
Sep 16 14:56:58.632141 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:56:58.632095    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q" podStartSLOduration=1.632079955 podStartE2EDuration="1.632079955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:56:58.631112355 +0000 UTC m=+1295.785190643" watchObservedRunningTime="2026-09-16 14:56:58.632079955 +0000 UTC m=+1295.786158243"
Sep 16 14:57:08.223961 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:08.223923    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:57:08.224398 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:08.223975    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:57:08.236087 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:08.236065    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:57:08.654066 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:08.654033    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:57:13.110983 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:13.110951    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"]
Sep 16 14:57:16.087574 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.087542    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"]
Sep 16 14:57:16.092343 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.092326    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 14:57:16.095393 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.095372    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d28bd5\""
Sep 16 14:57:16.095503 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.095430    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-d28bd5-dockercfg-lrjct\""
Sep 16 14:57:16.106994 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.106972    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"]
Sep 16 14:57:16.126471 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.126446    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqb7\" (UniqueName: \"kubernetes.io/projected/fb16db93-b470-4026-ad63-f4210fa03565-kube-api-access-vqqb7\") pod \"payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl\" (UID: \"fb16db93-b470-4026-ad63-f4210fa03565\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 14:57:16.126600 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.126503    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb16db93-b470-4026-ad63-f4210fa03565-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl\" (UID: \"fb16db93-b470-4026-ad63-f4210fa03565\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 14:57:16.227665 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.227628    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqb7\" (UniqueName: \"kubernetes.io/projected/fb16db93-b470-4026-ad63-f4210fa03565-kube-api-access-vqqb7\") pod \"payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl\" (UID: \"fb16db93-b470-4026-ad63-f4210fa03565\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 14:57:16.227837 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.227725    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb16db93-b470-4026-ad63-f4210fa03565-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl\" (UID: \"fb16db93-b470-4026-ad63-f4210fa03565\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 14:57:16.228429 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.228405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb16db93-b470-4026-ad63-f4210fa03565-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl\" (UID: \"fb16db93-b470-4026-ad63-f4210fa03565\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 14:57:16.236688 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.236659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqb7\" (UniqueName: \"kubernetes.io/projected/fb16db93-b470-4026-ad63-f4210fa03565-kube-api-access-vqqb7\") pod \"payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl\" (UID: \"fb16db93-b470-4026-ad63-f4210fa03565\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 14:57:16.402603 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.402517    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 14:57:16.681601 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:16.681573    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"]
Sep 16 14:57:16.683267 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:57:16.683241    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb16db93_b470_4026_ad63_f4210fa03565.slice/crio-37d75814498e8196f22bc34225d7734a089d1426d9570eb408a4ffc8c4b7abfb WatchSource:0}: Error finding container 37d75814498e8196f22bc34225d7734a089d1426d9570eb408a4ffc8c4b7abfb: Status 404 returned error can't find the container with id 37d75814498e8196f22bc34225d7734a089d1426d9570eb408a4ffc8c4b7abfb
Sep 16 14:57:17.693648 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:17.693614    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl" event={"ID":"fb16db93-b470-4026-ad63-f4210fa03565","Type":"ContainerStarted","Data":"673e9d63bf88afcbdf469c1182071b9e584ec2fa5c33639e37ee234d0c64ba97"}
Sep 16 14:57:17.694039 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:17.693669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl" event={"ID":"fb16db93-b470-4026-ad63-f4210fa03565","Type":"ContainerStarted","Data":"37d75814498e8196f22bc34225d7734a089d1426d9570eb408a4ffc8c4b7abfb"}
Sep 16 14:57:17.694039 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:17.693689    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 14:57:17.713158 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:17.713109    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl" podStartSLOduration=1.713095548 podStartE2EDuration="1.713095548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:57:17.711236503 +0000 UTC m=+1314.865314790" watchObservedRunningTime="2026-09-16 14:57:17.713095548 +0000 UTC m=+1314.867173834"
Sep 16 14:57:23.369468 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:23.369427    2558 scope.go:117] "RemoveContainer" containerID="de2c963b6763092f68376a61abdebdb9d0a288230a2759bf3673ba65c8384b7f"
Sep 16 14:57:28.130490 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.130448    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr" podUID="046a91ef-c653-425e-8f6a-1cb9758c5817" containerName="main" containerID="cri-o://23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33" gracePeriod=45
Sep 16 14:57:28.375588 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.375562    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:57:28.438156 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.438088    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046a91ef-c653-425e-8f6a-1cb9758c5817-tls-certs\") pod \"046a91ef-c653-425e-8f6a-1cb9758c5817\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") "
Sep 16 14:57:28.438156 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.438154    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-dshm\") pod \"046a91ef-c653-425e-8f6a-1cb9758c5817\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") "
Sep 16 14:57:28.438404 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.438200    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk6pq\" (UniqueName: \"kubernetes.io/projected/046a91ef-c653-425e-8f6a-1cb9758c5817-kube-api-access-rk6pq\") pod \"046a91ef-c653-425e-8f6a-1cb9758c5817\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") "
Sep 16 14:57:28.438404 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.438227    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-tmp-dir\") pod \"046a91ef-c653-425e-8f6a-1cb9758c5817\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") "
Sep 16 14:57:28.438404 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.438269    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-model-cache\") pod \"046a91ef-c653-425e-8f6a-1cb9758c5817\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") "
Sep 16 14:57:28.438404 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.438334    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-home\") pod \"046a91ef-c653-425e-8f6a-1cb9758c5817\" (UID: \"046a91ef-c653-425e-8f6a-1cb9758c5817\") "
Sep 16 14:57:28.438796 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.438668    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-model-cache" (OuterVolumeSpecName: "model-cache") pod "046a91ef-c653-425e-8f6a-1cb9758c5817" (UID: "046a91ef-c653-425e-8f6a-1cb9758c5817"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:57:28.438796 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.438764    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "046a91ef-c653-425e-8f6a-1cb9758c5817" (UID: "046a91ef-c653-425e-8f6a-1cb9758c5817"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:57:28.439000 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.438786    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-home" (OuterVolumeSpecName: "home") pod "046a91ef-c653-425e-8f6a-1cb9758c5817" (UID: "046a91ef-c653-425e-8f6a-1cb9758c5817"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:57:28.440499 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.440476    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046a91ef-c653-425e-8f6a-1cb9758c5817-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "046a91ef-c653-425e-8f6a-1cb9758c5817" (UID: "046a91ef-c653-425e-8f6a-1cb9758c5817"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:57:28.440772 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.440755    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046a91ef-c653-425e-8f6a-1cb9758c5817-kube-api-access-rk6pq" (OuterVolumeSpecName: "kube-api-access-rk6pq") pod "046a91ef-c653-425e-8f6a-1cb9758c5817" (UID: "046a91ef-c653-425e-8f6a-1cb9758c5817"). InnerVolumeSpecName "kube-api-access-rk6pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:57:28.440772 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.440761    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-dshm" (OuterVolumeSpecName: "dshm") pod "046a91ef-c653-425e-8f6a-1cb9758c5817" (UID: "046a91ef-c653-425e-8f6a-1cb9758c5817"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:57:28.539613 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.539585    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:57:28.539613 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.539611    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk6pq\" (UniqueName: \"kubernetes.io/projected/046a91ef-c653-425e-8f6a-1cb9758c5817-kube-api-access-rk6pq\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:57:28.539782 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.539622    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:57:28.539782 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.539631    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:57:28.539782 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.539639    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/046a91ef-c653-425e-8f6a-1cb9758c5817-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:57:28.539782 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.539646    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/046a91ef-c653-425e-8f6a-1cb9758c5817-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:57:28.701334 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.701262    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 14:57:28.752465 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.752442    2558 generic.go:356] "Generic (PLEG): container finished" podID="046a91ef-c653-425e-8f6a-1cb9758c5817" containerID="23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33" exitCode=0
Sep 16 14:57:28.752607 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.752481    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"
Sep 16 14:57:28.752607 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.752521    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr" event={"ID":"046a91ef-c653-425e-8f6a-1cb9758c5817","Type":"ContainerDied","Data":"23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33"}
Sep 16 14:57:28.752607 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.752557    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr" event={"ID":"046a91ef-c653-425e-8f6a-1cb9758c5817","Type":"ContainerDied","Data":"064e65bb32c5c3e09322f78a22719e94d117408c278557af4c6af8c3ace4ad37"}
Sep 16 14:57:28.752607 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.752573    2558 scope.go:117] "RemoveContainer" containerID="23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33"
Sep 16 14:57:28.767101 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.767080    2558 scope.go:117] "RemoveContainer" containerID="23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33"
Sep 16 14:57:28.767437 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:57:28.767416    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33\": container with ID starting with 23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33 not found: ID does not exist" containerID="23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33"
Sep 16 14:57:28.767506 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.767445    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33"} err="failed to get container status \"23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33\": rpc error: code = NotFound desc = could not find container \"23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33\": container with ID starting with 23d068d12fd551e2a0e1b9bec3922211d4fca13fa482033e26ebfa96ddcf9d33 not found: ID does not exist"
Sep 16 14:57:28.810430 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.810404    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"]
Sep 16 14:57:28.814679 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:28.814656    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-f4d89a/noten-model-f4d89a-kserve-578d7f599-kdssr"]
Sep 16 14:57:29.295805 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:29.295773    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046a91ef-c653-425e-8f6a-1cb9758c5817" path="/var/lib/kubelet/pods/046a91ef-c653-425e-8f6a-1cb9758c5817/volumes"
Sep 16 14:57:53.637741 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.637658    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"]
Sep 16 14:57:53.638132 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.638062    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046a91ef-c653-425e-8f6a-1cb9758c5817" containerName="main"
Sep 16 14:57:53.638132 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.638072    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="046a91ef-c653-425e-8f6a-1cb9758c5817" containerName="main"
Sep 16 14:57:53.638255 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.638149    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="046a91ef-c653-425e-8f6a-1cb9758c5817" containerName="main"
Sep 16 14:57:53.641198 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.641160    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.644467 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.644447    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-adb97937\"/\"default-dockercfg-dmpv2\""
Sep 16 14:57:53.645607 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.645579    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-adb97937\"/\"kube-root-ca.crt\""
Sep 16 14:57:53.645607 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.645598    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-adb97937\"/\"openshift-service-ca.crt\""
Sep 16 14:57:53.645766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.645616    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-adb97937\"/\"e2e-lifecycle-model-adb97937-kserve-self-signed-certs\""
Sep 16 14:57:53.665341 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.665313    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"]
Sep 16 14:57:53.685246 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.685219    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tls-certs\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.685385 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.685274    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-model-cache\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.685385 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.685308    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-dshm\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.685385 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.685347    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-home\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.685385 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.685368    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tmp-dir\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.685623 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.685397    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvv4f\" (UniqueName: \"kubernetes.io/projected/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-kube-api-access-kvv4f\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.785908 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.785875    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-model-cache\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.786085 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.785921    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-dshm\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.786085 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.785975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-home\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.786085 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.786007    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tmp-dir\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.786085 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.786058    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvv4f\" (UniqueName: \"kubernetes.io/projected/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-kube-api-access-kvv4f\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.786343 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.786114    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tls-certs\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.786402 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.786338    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-model-cache\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.786402 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.786367    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-home\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.786515 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.786488    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tmp-dir\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.788382 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.788360    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-dshm\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.788678 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.788662    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tls-certs\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.798643 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.798623    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvv4f\" (UniqueName: \"kubernetes.io/projected/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-kube-api-access-kvv4f\") pod \"e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") " pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:53.950626 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:53.950540    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:57:54.203416 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.203333    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"]
Sep 16 14:57:54.208011 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.207989    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 14:57:54.211611 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.211589    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-cadcf4\""
Sep 16 14:57:54.212413 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.212396    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-cadcf4-dockercfg-xjft7\""
Sep 16 14:57:54.236556 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.236455    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"]
Sep 16 14:57:54.245008 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.244986    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"]
Sep 16 14:57:54.247917 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:57:54.247891    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8f5f73e_2b75_4541_99c0_b4f572a41fb8.slice/crio-f7063650cd854d1f403209cd1a0fff734287688548a9711d6b30ad7f643d9133 WatchSource:0}: Error finding container f7063650cd854d1f403209cd1a0fff734287688548a9711d6b30ad7f643d9133: Status 404 returned error can't find the container with id f7063650cd854d1f403209cd1a0fff734287688548a9711d6b30ad7f643d9133
Sep 16 14:57:54.290187 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.290147    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19309040-11c1-4f60-870b-b60ea17febd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp\" (UID: \"19309040-11c1-4f60-870b-b60ea17febd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 14:57:54.290312 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.290297    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g224q\" (UniqueName: \"kubernetes.io/projected/19309040-11c1-4f60-870b-b60ea17febd5-kube-api-access-g224q\") pod \"payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp\" (UID: \"19309040-11c1-4f60-870b-b60ea17febd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 14:57:54.349676 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.349643    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"]
Sep 16 14:57:54.353221 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.353202    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 14:57:54.375044 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.375009    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"]
Sep 16 14:57:54.390938 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.390910    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwtt\" (UniqueName: \"kubernetes.io/projected/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-kube-api-access-pkwtt\") pod \"payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs\" (UID: \"af85bb43-ea5c-4233-bee0-d597a4d6b8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 14:57:54.391115 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.390947    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g224q\" (UniqueName: \"kubernetes.io/projected/19309040-11c1-4f60-870b-b60ea17febd5-kube-api-access-g224q\") pod \"payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp\" (UID: \"19309040-11c1-4f60-870b-b60ea17febd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 14:57:54.391115 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.391061    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19309040-11c1-4f60-870b-b60ea17febd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp\" (UID: \"19309040-11c1-4f60-870b-b60ea17febd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 14:57:54.391259 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.391157    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs\" (UID: \"af85bb43-ea5c-4233-bee0-d597a4d6b8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 14:57:54.391754 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.391738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19309040-11c1-4f60-870b-b60ea17febd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp\" (UID: \"19309040-11c1-4f60-870b-b60ea17febd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 14:57:54.404410 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.404379    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g224q\" (UniqueName: \"kubernetes.io/projected/19309040-11c1-4f60-870b-b60ea17febd5-kube-api-access-g224q\") pod \"payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp\" (UID: \"19309040-11c1-4f60-870b-b60ea17febd5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 14:57:54.492202 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.492087    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwtt\" (UniqueName: \"kubernetes.io/projected/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-kube-api-access-pkwtt\") pod \"payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs\" (UID: \"af85bb43-ea5c-4233-bee0-d597a4d6b8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 14:57:54.492363 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.492226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs\" (UID: \"af85bb43-ea5c-4233-bee0-d597a4d6b8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 14:57:54.492855 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.492835    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs\" (UID: \"af85bb43-ea5c-4233-bee0-d597a4d6b8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 14:57:54.504436 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.504413    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwtt\" (UniqueName: \"kubernetes.io/projected/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-kube-api-access-pkwtt\") pod \"payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs\" (UID: \"af85bb43-ea5c-4233-bee0-d597a4d6b8d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 14:57:54.519522 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.519491    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 14:57:54.664641 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.664605    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 14:57:54.797700 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.797672    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"]
Sep 16 14:57:54.798034 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:57:54.798005    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19309040_11c1_4f60_870b_b60ea17febd5.slice/crio-b6466e47038ad5abec8b49522785194f20fe4a181539931bf3ec1edf7f946a41 WatchSource:0}: Error finding container b6466e47038ad5abec8b49522785194f20fe4a181539931bf3ec1edf7f946a41: Status 404 returned error can't find the container with id b6466e47038ad5abec8b49522785194f20fe4a181539931bf3ec1edf7f946a41
Sep 16 14:57:54.895009 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.894949    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp" event={"ID":"19309040-11c1-4f60-870b-b60ea17febd5","Type":"ContainerStarted","Data":"1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759"}
Sep 16 14:57:54.895201 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.895035    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp" event={"ID":"19309040-11c1-4f60-870b-b60ea17febd5","Type":"ContainerStarted","Data":"b6466e47038ad5abec8b49522785194f20fe4a181539931bf3ec1edf7f946a41"}
Sep 16 14:57:54.895201 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.895069    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 14:57:54.899932 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.897279    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx" event={"ID":"f8f5f73e-2b75-4541-99c0-b4f572a41fb8","Type":"ContainerStarted","Data":"e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b"}
Sep 16 14:57:54.899932 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.897341    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx" event={"ID":"f8f5f73e-2b75-4541-99c0-b4f572a41fb8","Type":"ContainerStarted","Data":"f7063650cd854d1f403209cd1a0fff734287688548a9711d6b30ad7f643d9133"}
Sep 16 14:57:54.942966 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.942910    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp" podStartSLOduration=0.94289452 podStartE2EDuration="942.89452ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:57:54.940187022 +0000 UTC m=+1352.094265301" watchObservedRunningTime="2026-09-16 14:57:54.94289452 +0000 UTC m=+1352.096972803"
Sep 16 14:57:54.974994 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.974964    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"]
Sep 16 14:57:54.977706 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:57:54.977668    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf85bb43_ea5c_4233_bee0_d597a4d6b8d3.slice/crio-6461d002938cc1d903744b8da42a9170c60cbbde108d5d0b9973278d0ee722db WatchSource:0}: Error finding container 6461d002938cc1d903744b8da42a9170c60cbbde108d5d0b9973278d0ee722db: Status 404 returned error can't find the container with id 6461d002938cc1d903744b8da42a9170c60cbbde108d5d0b9973278d0ee722db
Sep 16 14:57:54.978097 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:54.978060    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx" podStartSLOduration=1.978046805 podStartE2EDuration="1.978046805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:57:54.975617974 +0000 UTC m=+1352.129696258" watchObservedRunningTime="2026-09-16 14:57:54.978046805 +0000 UTC m=+1352.132125279"
Sep 16 14:57:55.904878 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:55.904841    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs" event={"ID":"af85bb43-ea5c-4233-bee0-d597a4d6b8d3","Type":"ContainerStarted","Data":"3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6"}
Sep 16 14:57:55.905330 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:55.904909    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs" event={"ID":"af85bb43-ea5c-4233-bee0-d597a4d6b8d3","Type":"ContainerStarted","Data":"6461d002938cc1d903744b8da42a9170c60cbbde108d5d0b9973278d0ee722db"}
Sep 16 14:57:55.905330 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:55.905192    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 14:57:55.925389 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:57:55.925341    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs" podStartSLOduration=1.9253261990000001 podStartE2EDuration="1.925326199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:57:55.923029386 +0000 UTC m=+1353.077107675" watchObservedRunningTime="2026-09-16 14:57:55.925326199 +0000 UTC m=+1353.079404486"
Sep 16 14:58:03.950947 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:03.950917    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:58:03.951396 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:03.950966    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:58:03.963818 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:03.963795    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:58:04.966107 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:04.966075    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:58:05.906720 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:05.906691    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 14:58:06.913736 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:06.913700    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 14:58:23.409052 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:23.409027    2558 scope.go:117] "RemoveContainer" containerID="120c30939dbbe8932b1ec3e8f34c31c005ea2e73cae6b46a66876a9964a694b4"
Sep 16 14:58:23.417183 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:23.417150    2558 scope.go:117] "RemoveContainer" containerID="7eda4b306405fad3397167aaa8f98eb31df227d76f119dbb886b71b673d73c65"
Sep 16 14:58:23.425197 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:23.425179    2558 scope.go:117] "RemoveContainer" containerID="6b37cfc5496f255226be2b77c5343a148c8c4e6dd2ecf950ba11dc50eb4a9ece"
Sep 16 14:58:26.076347 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:26.076317    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"]
Sep 16 14:58:41.095157 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.095115    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q" podUID="d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" containerName="main" containerID="cri-o://39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089" gracePeriod=45
Sep 16 14:58:41.341120 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.341096    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:58:41.452265 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452156    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tmp-dir\") pod \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") "
Sep 16 14:58:41.452265 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452235    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-home\") pod \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") "
Sep 16 14:58:41.452265 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452265    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tls-certs\") pod \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") "
Sep 16 14:58:41.452553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452310    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49nfp\" (UniqueName: \"kubernetes.io/projected/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-kube-api-access-49nfp\") pod \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") "
Sep 16 14:58:41.452553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452395    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-dshm\") pod \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") "
Sep 16 14:58:41.452553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452405    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" (UID: "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:58:41.452553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452418    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-model-cache\") pod \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\" (UID: \"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d\") "
Sep 16 14:58:41.452731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452537    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-home" (OuterVolumeSpecName: "home") pod "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" (UID: "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:58:41.452731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452609    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" (UID: "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:58:41.452826 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452809    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:41.452873 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452830    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:41.452873 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.452841    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:41.454637 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.454612    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-kube-api-access-49nfp" (OuterVolumeSpecName: "kube-api-access-49nfp") pod "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" (UID: "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d"). InnerVolumeSpecName "kube-api-access-49nfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:58:41.454726 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.454672    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" (UID: "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:58:41.454726 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.454679    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-dshm" (OuterVolumeSpecName: "dshm") pod "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" (UID: "d97fa8f1-1be2-4088-8dd4-92fc90f18c5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:58:41.553700 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.553665    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:41.553700 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.553694    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49nfp\" (UniqueName: \"kubernetes.io/projected/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-kube-api-access-49nfp\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:41.553700 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:41.553705    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:42.156046 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:42.156011    2558 generic.go:356] "Generic (PLEG): container finished" podID="d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" containerID="39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089" exitCode=0
Sep 16 14:58:42.156553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:42.156068    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"
Sep 16 14:58:42.156553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:42.156097    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q" event={"ID":"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d","Type":"ContainerDied","Data":"39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089"}
Sep 16 14:58:42.156553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:42.156149    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q" event={"ID":"d97fa8f1-1be2-4088-8dd4-92fc90f18c5d","Type":"ContainerDied","Data":"e77b5303ead89fe4e484975802071271599b4d7c75184f40c7be03d6ada8d791"}
Sep 16 14:58:42.156553 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:42.156187    2558 scope.go:117] "RemoveContainer" containerID="39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089"
Sep 16 14:58:42.165593 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:42.165573    2558 scope.go:117] "RemoveContainer" containerID="39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089"
Sep 16 14:58:42.165867 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:58:42.165849    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089\": container with ID starting with 39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089 not found: ID does not exist" containerID="39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089"
Sep 16 14:58:42.165935 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:42.165878    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089"} err="failed to get container status \"39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089\": rpc error: code = NotFound desc = could not find container \"39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089\": container with ID starting with 39c1db8d283bb0d1b39613670c98f49af25114a3133ff6c4a79ac51fb284a089 not found: ID does not exist"
Sep 16 14:58:42.181425 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:42.181397    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"]
Sep 16 14:58:42.185680 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:42.185655    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-b-b025547b-kserve-f8b7fcb58-4l57q"]
Sep 16 14:58:43.296842 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:43.296812    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" path="/var/lib/kubelet/pods/d97fa8f1-1be2-4088-8dd4-92fc90f18c5d/volumes"
Sep 16 14:58:43.387252 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:43.387212    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"]
Sep 16 14:58:51.743754 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.743716    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"]
Sep 16 14:58:51.744240 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.744223    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" containerName="main"
Sep 16 14:58:51.744298 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.744243    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" containerName="main"
Sep 16 14:58:51.744347 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.744317    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97fa8f1-1be2-4088-8dd4-92fc90f18c5d" containerName="main"
Sep 16 14:58:51.748743 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.748723    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.754249 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.754228    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d28bd5\"/\"default-dockercfg-m47zc\""
Sep 16 14:58:51.754488 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.754464    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d28bd5\"/\"openshift-service-ca.crt\""
Sep 16 14:58:51.754588 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.754499    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d28bd5\"/\"kube-root-ca.crt\""
Sep 16 14:58:51.754647 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.754595    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d28bd5\"/\"auth-test-model-d28bd5-kserve-self-signed-certs\""
Sep 16 14:58:51.769135 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.769099    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"]
Sep 16 14:58:51.868969 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.868927    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-tmp-dir\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.868969 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.868966    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-model-cache\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.869231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.869044    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e706e0-4c29-4109-aa90-a764d78acefb-tls-certs\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.869231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.869123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-dshm\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.869231 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.869148    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lgnr\" (UniqueName: \"kubernetes.io/projected/00e706e0-4c29-4109-aa90-a764d78acefb-kube-api-access-2lgnr\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.869351 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.869262    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-home\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.970060 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.970021    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-tmp-dir\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.970060 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.970060    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-model-cache\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.970332 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.970100    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e706e0-4c29-4109-aa90-a764d78acefb-tls-certs\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.970332 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.970130    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-dshm\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.970332 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.970149    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lgnr\" (UniqueName: \"kubernetes.io/projected/00e706e0-4c29-4109-aa90-a764d78acefb-kube-api-access-2lgnr\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.970332 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.970270    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-home\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.970551 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.970462    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-tmp-dir\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.970612 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.970563    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-home\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.970666 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.970618    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-model-cache\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.972414 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.972393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-dshm\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.972538 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.972519    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e706e0-4c29-4109-aa90-a764d78acefb-tls-certs\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:51.979274 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:51.979247    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lgnr\" (UniqueName: \"kubernetes.io/projected/00e706e0-4c29-4109-aa90-a764d78acefb-kube-api-access-2lgnr\") pod \"auth-test-model-d28bd5-kserve-c6764d95f-mlttf\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:52.058213 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:52.058177    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:58:52.335424 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:52.335401    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"]
Sep 16 14:58:52.337454 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:58:52.337427    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00e706e0_4c29_4109_aa90_a764d78acefb.slice/crio-522429ab6527e5dc3b1938f485077c911d806f6bde7632c29879eb0adbe42a17 WatchSource:0}: Error finding container 522429ab6527e5dc3b1938f485077c911d806f6bde7632c29879eb0adbe42a17: Status 404 returned error can't find the container with id 522429ab6527e5dc3b1938f485077c911d806f6bde7632c29879eb0adbe42a17
Sep 16 14:58:53.222263 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:53.222228    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf" event={"ID":"00e706e0-4c29-4109-aa90-a764d78acefb","Type":"ContainerStarted","Data":"eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0"}
Sep 16 14:58:53.222263 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:53.222267    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf" event={"ID":"00e706e0-4c29-4109-aa90-a764d78acefb","Type":"ContainerStarted","Data":"522429ab6527e5dc3b1938f485077c911d806f6bde7632c29879eb0adbe42a17"}
Sep 16 14:58:53.240455 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:53.240414    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf" podStartSLOduration=2.240397867 podStartE2EDuration="2.240397867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:58:53.239577838 +0000 UTC m=+1410.393656128" watchObservedRunningTime="2026-09-16 14:58:53.240397867 +0000 UTC m=+1410.394476154"
Sep 16 14:58:58.406186 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.406136    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5" podUID="a234d648-d7f9-4d31-aca6-6ce8c22384a7" containerName="main" containerID="cri-o://78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290" gracePeriod=45
Sep 16 14:58:58.660788 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.660733    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:58:58.835721 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.835693    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-model-cache\") pod \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") "
Sep 16 14:58:58.835721 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.835726    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hv5j\" (UniqueName: \"kubernetes.io/projected/a234d648-d7f9-4d31-aca6-6ce8c22384a7-kube-api-access-7hv5j\") pod \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") "
Sep 16 14:58:58.835974 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.835765    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tls-certs\") pod \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") "
Sep 16 14:58:58.835974 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.835819    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-home\") pod \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") "
Sep 16 14:58:58.835974 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.835838    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-dshm\") pod \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") "
Sep 16 14:58:58.835974 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.835866    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tmp-dir\") pod \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\" (UID: \"a234d648-d7f9-4d31-aca6-6ce8c22384a7\") "
Sep 16 14:58:58.836218 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.836025    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "a234d648-d7f9-4d31-aca6-6ce8c22384a7" (UID: "a234d648-d7f9-4d31-aca6-6ce8c22384a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:58:58.836218 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.836130    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-home" (OuterVolumeSpecName: "home") pod "a234d648-d7f9-4d31-aca6-6ce8c22384a7" (UID: "a234d648-d7f9-4d31-aca6-6ce8c22384a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:58:58.836320 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.836245    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a234d648-d7f9-4d31-aca6-6ce8c22384a7" (UID: "a234d648-d7f9-4d31-aca6-6ce8c22384a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:58:58.836376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.836314    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:58.836376 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.836335    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:58.837970 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.837945    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-dshm" (OuterVolumeSpecName: "dshm") pod "a234d648-d7f9-4d31-aca6-6ce8c22384a7" (UID: "a234d648-d7f9-4d31-aca6-6ce8c22384a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:58:58.838071 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.837964    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a234d648-d7f9-4d31-aca6-6ce8c22384a7" (UID: "a234d648-d7f9-4d31-aca6-6ce8c22384a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:58:58.838071 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.837986    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a234d648-d7f9-4d31-aca6-6ce8c22384a7-kube-api-access-7hv5j" (OuterVolumeSpecName: "kube-api-access-7hv5j") pod "a234d648-d7f9-4d31-aca6-6ce8c22384a7" (UID: "a234d648-d7f9-4d31-aca6-6ce8c22384a7"). InnerVolumeSpecName "kube-api-access-7hv5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:58:58.937554 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.937480    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hv5j\" (UniqueName: \"kubernetes.io/projected/a234d648-d7f9-4d31-aca6-6ce8c22384a7-kube-api-access-7hv5j\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:58.937554 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.937509    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:58.937554 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.937518    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:58.937554 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:58.937526    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a234d648-d7f9-4d31-aca6-6ce8c22384a7-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:58:59.257590 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:59.257508    2558 generic.go:356] "Generic (PLEG): container finished" podID="a234d648-d7f9-4d31-aca6-6ce8c22384a7" containerID="78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290" exitCode=0
Sep 16 14:58:59.257590 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:59.257555    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"
Sep 16 14:58:59.257800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:59.257586    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5" event={"ID":"a234d648-d7f9-4d31-aca6-6ce8c22384a7","Type":"ContainerDied","Data":"78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290"}
Sep 16 14:58:59.257800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:59.257628    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5" event={"ID":"a234d648-d7f9-4d31-aca6-6ce8c22384a7","Type":"ContainerDied","Data":"9902f63c0af23b4cda8ad26198ebad7c5240ffbd5228d13a08c87315de8da2bd"}
Sep 16 14:58:59.257800 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:59.257646    2558 scope.go:117] "RemoveContainer" containerID="78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290"
Sep 16 14:58:59.266826 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:59.266810    2558 scope.go:117] "RemoveContainer" containerID="78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290"
Sep 16 14:58:59.267096 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:58:59.267076    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290\": container with ID starting with 78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290 not found: ID does not exist" containerID="78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290"
Sep 16 14:58:59.267196 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:59.267104    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290"} err="failed to get container status \"78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290\": rpc error: code = NotFound desc = could not find container \"78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290\": container with ID starting with 78b35225b6f36e175bf03e907ce7f615b1f5f5c5c396f75520b126bccd76d290 not found: ID does not exist"
Sep 16 14:58:59.296820 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:59.296794    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"]
Sep 16 14:58:59.296820 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:58:59.296827    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/e2e-conflict-a-b025547b-kserve-9f7bd5c48-dwqc5"]
Sep 16 14:59:01.295588 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:01.295555    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a234d648-d7f9-4d31-aca6-6ce8c22384a7" path="/var/lib/kubelet/pods/a234d648-d7f9-4d31-aca6-6ce8c22384a7/volumes"
Sep 16 14:59:02.058479 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.058438    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:59:02.058716 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.058693    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:59:02.071030 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.071001    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:59:02.286237 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.286133    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 14:59:02.356986 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.356958    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"]
Sep 16 14:59:02.369931 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.369900    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a234d648-d7f9-4d31-aca6-6ce8c22384a7" containerName="main"
Sep 16 14:59:02.370148 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.370134    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a234d648-d7f9-4d31-aca6-6ce8c22384a7" containerName="main"
Sep 16 14:59:02.370557 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.370539    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a234d648-d7f9-4d31-aca6-6ce8c22384a7" containerName="main"
Sep 16 14:59:02.377299 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.377261    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"]
Sep 16 14:59:02.377426 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.377403    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"
Sep 16 14:59:02.380085 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.380068    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 14:59:02.471270 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.471237    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0313be92-c74e-4e58-aead-d5593567026d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-adb97937-w9vjr\" (UID: \"0313be92-c74e-4e58-aead-d5593567026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"
Sep 16 14:59:02.571931 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.571889    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0313be92-c74e-4e58-aead-d5593567026d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-adb97937-w9vjr\" (UID: \"0313be92-c74e-4e58-aead-d5593567026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"
Sep 16 14:59:02.572575 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.572555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0313be92-c74e-4e58-aead-d5593567026d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-adb97937-w9vjr\" (UID: \"0313be92-c74e-4e58-aead-d5593567026d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"
Sep 16 14:59:02.688688 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.688641    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"
Sep 16 14:59:02.967935 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:02.967912    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"]
Sep 16 14:59:02.970274 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:59:02.970246    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0313be92_c74e_4e58_aead_d5593567026d.slice/crio-bb1ffeaa1f51cde511f9db4fcc6acd29215c63616d4628eff27dd2837230ef59 WatchSource:0}: Error finding container bb1ffeaa1f51cde511f9db4fcc6acd29215c63616d4628eff27dd2837230ef59: Status 404 returned error can't find the container with id bb1ffeaa1f51cde511f9db4fcc6acd29215c63616d4628eff27dd2837230ef59
Sep 16 14:59:03.281463 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:03.281432    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr" event={"ID":"0313be92-c74e-4e58-aead-d5593567026d","Type":"ContainerStarted","Data":"dabc262e561161571f5e1bd3f852ca8b978418f5773a2570eb382d3ce10b6268"}
Sep 16 14:59:03.281612 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:03.281483    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr" event={"ID":"0313be92-c74e-4e58-aead-d5593567026d","Type":"ContainerStarted","Data":"bb1ffeaa1f51cde511f9db4fcc6acd29215c63616d4628eff27dd2837230ef59"}
Sep 16 14:59:03.300357 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:03.300310    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr" podStartSLOduration=1.300295009 podStartE2EDuration="1.300295009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:59:03.299320822 +0000 UTC m=+1420.453399110" watchObservedRunningTime="2026-09-16 14:59:03.300295009 +0000 UTC m=+1420.454373296"
Sep 16 14:59:04.288599 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:04.288562    2558 generic.go:356] "Generic (PLEG): container finished" podID="0313be92-c74e-4e58-aead-d5593567026d" containerID="dabc262e561161571f5e1bd3f852ca8b978418f5773a2570eb382d3ce10b6268" exitCode=0
Sep 16 14:59:04.289104 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:04.288660    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr" event={"ID":"0313be92-c74e-4e58-aead-d5593567026d","Type":"ContainerDied","Data":"dabc262e561161571f5e1bd3f852ca8b978418f5773a2570eb382d3ce10b6268"}
Sep 16 14:59:05.428323 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:05.428299    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"
Sep 16 14:59:05.499953 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:05.499923    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0313be92-c74e-4e58-aead-d5593567026d-maas-api-service-ca\") pod \"0313be92-c74e-4e58-aead-d5593567026d\" (UID: \"0313be92-c74e-4e58-aead-d5593567026d\") "
Sep 16 14:59:05.500285 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:05.500266    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0313be92-c74e-4e58-aead-d5593567026d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0313be92-c74e-4e58-aead-d5593567026d" (UID: "0313be92-c74e-4e58-aead-d5593567026d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:59:05.601509 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:05.601479    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0313be92-c74e-4e58-aead-d5593567026d-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:59:06.301501 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:06.301471    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr" event={"ID":"0313be92-c74e-4e58-aead-d5593567026d","Type":"ContainerDied","Data":"bb1ffeaa1f51cde511f9db4fcc6acd29215c63616d4628eff27dd2837230ef59"}
Sep 16 14:59:06.310020 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:06.301811    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb1ffeaa1f51cde511f9db4fcc6acd29215c63616d4628eff27dd2837230ef59"
Sep 16 14:59:06.310020 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:06.301850    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"
Sep 16 14:59:18.037210 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.037121    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"]
Sep 16 14:59:18.037737 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.037345    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9" podUID="1c897f19-ca8b-4262-b862-2bbeea724281" containerName="payload-processing" containerID="cri-o://393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b" gracePeriod=30
Sep 16 14:59:18.045389 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.045360    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"]
Sep 16 14:59:18.045616 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.045590    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd" podUID="8ead3d07-f374-4f6b-bcbf-d32502009563" containerName="payload-pre-processing" containerID="cri-o://0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d" gracePeriod=30
Sep 16 14:59:18.291386 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.291294    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"]
Sep 16 14:59:18.293745 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.293720    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-adb97937-w9vjr"]
Sep 16 14:59:18.335893 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.335867    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:59:18.337906 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.337884    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:59:18.362936 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.362415    2558 generic.go:356] "Generic (PLEG): container finished" podID="8ead3d07-f374-4f6b-bcbf-d32502009563" containerID="0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d" exitCode=0
Sep 16 14:59:18.362936 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.362468    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"
Sep 16 14:59:18.362936 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.362480    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd" event={"ID":"8ead3d07-f374-4f6b-bcbf-d32502009563","Type":"ContainerDied","Data":"0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d"}
Sep 16 14:59:18.362936 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.362553    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd" event={"ID":"8ead3d07-f374-4f6b-bcbf-d32502009563","Type":"ContainerDied","Data":"7a9f74cb9f67311f071ee090b5aed454de3ada26af746a2e7dfb0d944acc1d8e"}
Sep 16 14:59:18.362936 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.362585    2558 scope.go:117] "RemoveContainer" containerID="0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d"
Sep 16 14:59:18.370391 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.370363    2558 generic.go:356] "Generic (PLEG): container finished" podID="1c897f19-ca8b-4262-b862-2bbeea724281" containerID="393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b" exitCode=0
Sep 16 14:59:18.370558 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.370540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9" event={"ID":"1c897f19-ca8b-4262-b862-2bbeea724281","Type":"ContainerDied","Data":"393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b"}
Sep 16 14:59:18.370715 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.370701    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9" event={"ID":"1c897f19-ca8b-4262-b862-2bbeea724281","Type":"ContainerDied","Data":"02b3448bfed7dd3bd19ef1c62719240a232b4dbc87f815a89ea321be388293ca"}
Sep 16 14:59:18.370903 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.370887    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"
Sep 16 14:59:18.385025 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.385001    2558 scope.go:117] "RemoveContainer" containerID="0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d"
Sep 16 14:59:18.385324 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:59:18.385302    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d\": container with ID starting with 0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d not found: ID does not exist" containerID="0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d"
Sep 16 14:59:18.385426 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.385330    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d"} err="failed to get container status \"0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d\": rpc error: code = NotFound desc = could not find container \"0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d\": container with ID starting with 0ab4578bebf01c375737f6492cc980e7192edb9c0988abc163223b9c9a73406d not found: ID does not exist"
Sep 16 14:59:18.385426 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.385346    2558 scope.go:117] "RemoveContainer" containerID="393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b"
Sep 16 14:59:18.396086 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.396045    2558 scope.go:117] "RemoveContainer" containerID="393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b"
Sep 16 14:59:18.396526 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:59:18.396501    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b\": container with ID starting with 393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b not found: ID does not exist" containerID="393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b"
Sep 16 14:59:18.396646 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.396537    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b"} err="failed to get container status \"393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b\": rpc error: code = NotFound desc = could not find container \"393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b\": container with ID starting with 393c974d93cdc5e09c55fcd53043b0ebaf2ecf442895688bcb1ced3bf19e0b1b not found: ID does not exist"
Sep 16 14:59:18.424638 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.424614    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phbxl\" (UniqueName: \"kubernetes.io/projected/8ead3d07-f374-4f6b-bcbf-d32502009563-kube-api-access-phbxl\") pod \"8ead3d07-f374-4f6b-bcbf-d32502009563\" (UID: \"8ead3d07-f374-4f6b-bcbf-d32502009563\") "
Sep 16 14:59:18.424766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.424680    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ead3d07-f374-4f6b-bcbf-d32502009563-plugins-config-volume\") pod \"8ead3d07-f374-4f6b-bcbf-d32502009563\" (UID: \"8ead3d07-f374-4f6b-bcbf-d32502009563\") "
Sep 16 14:59:18.424766 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.424716    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czwfp\" (UniqueName: \"kubernetes.io/projected/1c897f19-ca8b-4262-b862-2bbeea724281-kube-api-access-czwfp\") pod \"1c897f19-ca8b-4262-b862-2bbeea724281\" (UID: \"1c897f19-ca8b-4262-b862-2bbeea724281\") "
Sep 16 14:59:18.424883 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.424777    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c897f19-ca8b-4262-b862-2bbeea724281-plugins-config-volume\") pod \"1c897f19-ca8b-4262-b862-2bbeea724281\" (UID: \"1c897f19-ca8b-4262-b862-2bbeea724281\") "
Sep 16 14:59:18.425111 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.425085    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ead3d07-f374-4f6b-bcbf-d32502009563-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ead3d07-f374-4f6b-bcbf-d32502009563" (UID: "8ead3d07-f374-4f6b-bcbf-d32502009563"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:59:18.425234 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.425207    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c897f19-ca8b-4262-b862-2bbeea724281-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c897f19-ca8b-4262-b862-2bbeea724281" (UID: "1c897f19-ca8b-4262-b862-2bbeea724281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:59:18.426795 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.426772    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ead3d07-f374-4f6b-bcbf-d32502009563-kube-api-access-phbxl" (OuterVolumeSpecName: "kube-api-access-phbxl") pod "8ead3d07-f374-4f6b-bcbf-d32502009563" (UID: "8ead3d07-f374-4f6b-bcbf-d32502009563"). InnerVolumeSpecName "kube-api-access-phbxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:59:18.426886 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.426795    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c897f19-ca8b-4262-b862-2bbeea724281-kube-api-access-czwfp" (OuterVolumeSpecName: "kube-api-access-czwfp") pod "1c897f19-ca8b-4262-b862-2bbeea724281" (UID: "1c897f19-ca8b-4262-b862-2bbeea724281"). InnerVolumeSpecName "kube-api-access-czwfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:59:18.526576 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.526544    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phbxl\" (UniqueName: \"kubernetes.io/projected/8ead3d07-f374-4f6b-bcbf-d32502009563-kube-api-access-phbxl\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:59:18.526576 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.526571    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ead3d07-f374-4f6b-bcbf-d32502009563-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:59:18.526576 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.526580    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czwfp\" (UniqueName: \"kubernetes.io/projected/1c897f19-ca8b-4262-b862-2bbeea724281-kube-api-access-czwfp\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:59:18.526792 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.526589    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c897f19-ca8b-4262-b862-2bbeea724281-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:59:18.701568 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.701542    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"]
Sep 16 14:59:18.711206 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.711184    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-adb97937-8644758594-68ftd"]
Sep 16 14:59:18.726217 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.726161    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"]
Sep 16 14:59:18.733229 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:18.733208    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-adb97937-7f847ff7b9-qmkj9"]
Sep 16 14:59:19.296241 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:19.296209    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0313be92-c74e-4e58-aead-d5593567026d" path="/var/lib/kubelet/pods/0313be92-c74e-4e58-aead-d5593567026d/volumes"
Sep 16 14:59:19.296602 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:19.296585    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c897f19-ca8b-4262-b862-2bbeea724281" path="/var/lib/kubelet/pods/1c897f19-ca8b-4262-b862-2bbeea724281/volumes"
Sep 16 14:59:19.296895 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:19.296883    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ead3d07-f374-4f6b-bcbf-d32502009563" path="/var/lib/kubelet/pods/8ead3d07-f374-4f6b-bcbf-d32502009563/volumes"
Sep 16 14:59:26.416731 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:26.416694    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"]
Sep 16 14:59:38.603343 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.603306    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"]
Sep 16 14:59:38.603737 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.603712    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ead3d07-f374-4f6b-bcbf-d32502009563" containerName="payload-pre-processing"
Sep 16 14:59:38.603737 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.603724    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ead3d07-f374-4f6b-bcbf-d32502009563" containerName="payload-pre-processing"
Sep 16 14:59:38.603737 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.603734    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0313be92-c74e-4e58-aead-d5593567026d" containerName="revoke-keys"
Sep 16 14:59:38.603737 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.603740    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0313be92-c74e-4e58-aead-d5593567026d" containerName="revoke-keys"
Sep 16 14:59:38.603869 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.603755    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c897f19-ca8b-4262-b862-2bbeea724281" containerName="payload-processing"
Sep 16 14:59:38.603869 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.603761    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c897f19-ca8b-4262-b862-2bbeea724281" containerName="payload-processing"
Sep 16 14:59:38.603869 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.603827    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c897f19-ca8b-4262-b862-2bbeea724281" containerName="payload-processing"
Sep 16 14:59:38.603869 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.603837    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0313be92-c74e-4e58-aead-d5593567026d" containerName="revoke-keys"
Sep 16 14:59:38.603869 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.603843    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ead3d07-f374-4f6b-bcbf-d32502009563" containerName="payload-pre-processing"
Sep 16 14:59:38.606569 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.606553    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.611785 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.611760    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cadcf4\"/\"openshift-service-ca.crt\""
Sep 16 14:59:38.611921 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.611852    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cadcf4\"/\"kube-root-ca.crt\""
Sep 16 14:59:38.612961 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.612935    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cadcf4\"/\"default-dockercfg-r6t6g\""
Sep 16 14:59:38.613098 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.612988    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cadcf4\"/\"auth-test-model-cadcf4-kserve-self-signed-certs\""
Sep 16 14:59:38.625123 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.625102    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2p4m\" (UniqueName: \"kubernetes.io/projected/88d4f8a8-846e-4a37-937a-7a978e8f9e24-kube-api-access-d2p4m\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.625242 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.625129    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tmp-dir\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.625242 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.625151    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-dshm\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.625242 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.625191    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tls-certs\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.625425 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.625264    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-home\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.625425 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.625388    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-model-cache\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.638955 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.638931    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"]
Sep 16 14:59:38.726112 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.726079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2p4m\" (UniqueName: \"kubernetes.io/projected/88d4f8a8-846e-4a37-937a-7a978e8f9e24-kube-api-access-d2p4m\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.726308 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.726117    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tmp-dir\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.726308 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.726143    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-dshm\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.726308 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.726198    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tls-certs\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.726308 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.726232    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-home\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.726556 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.726359    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-model-cache\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.726615 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.726554    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tmp-dir\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.726663 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.726633    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-model-cache\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.726703 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.726683    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-home\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.728582 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.728561    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-dshm\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.728681 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.728664    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tls-certs\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.741272 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.741249    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2p4m\" (UniqueName: \"kubernetes.io/projected/88d4f8a8-846e-4a37-937a-7a978e8f9e24-kube-api-access-d2p4m\") pod \"auth-test-model-cadcf4-kserve-8476756df9-ch2hw\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:38.916442 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:38.916354    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:39.193809 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:39.193783    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"]
Sep 16 14:59:39.196068 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:59:39.196041    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88d4f8a8_846e_4a37_937a_7a978e8f9e24.slice/crio-99a93859c9fd232ceccf9757bc37961d24a80e4943cd40833b3ab10a9ee48e08 WatchSource:0}: Error finding container 99a93859c9fd232ceccf9757bc37961d24a80e4943cd40833b3ab10a9ee48e08: Status 404 returned error can't find the container with id 99a93859c9fd232ceccf9757bc37961d24a80e4943cd40833b3ab10a9ee48e08
Sep 16 14:59:39.484019 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:39.483928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw" event={"ID":"88d4f8a8-846e-4a37-937a-7a978e8f9e24","Type":"ContainerStarted","Data":"5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7"}
Sep 16 14:59:39.484019 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:39.483975    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw" event={"ID":"88d4f8a8-846e-4a37-937a-7a978e8f9e24","Type":"ContainerStarted","Data":"99a93859c9fd232ceccf9757bc37961d24a80e4943cd40833b3ab10a9ee48e08"}
Sep 16 14:59:41.436154 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.436096    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx" podUID="f8f5f73e-2b75-4541-99c0-b4f572a41fb8" containerName="main" containerID="cri-o://e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b" gracePeriod=45
Sep 16 14:59:41.697215 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.697127    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:59:41.716368 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.716306    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw" podStartSLOduration=3.716288624 podStartE2EDuration="3.716288624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:59:39.506018355 +0000 UTC m=+1456.660096643" watchObservedRunningTime="2026-09-16 14:59:41.716288624 +0000 UTC m=+1458.870366910"
Sep 16 14:59:41.752738 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.752705    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvv4f\" (UniqueName: \"kubernetes.io/projected/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-kube-api-access-kvv4f\") pod \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") "
Sep 16 14:59:41.752941 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.752774    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tmp-dir\") pod \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") "
Sep 16 14:59:41.752941 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.752841    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tls-certs\") pod \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") "
Sep 16 14:59:41.752941 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.752913    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-model-cache\") pod \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") "
Sep 16 14:59:41.753128 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.752974    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-dshm\") pod \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") "
Sep 16 14:59:41.753128 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.753015    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-home\") pod \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\" (UID: \"f8f5f73e-2b75-4541-99c0-b4f572a41fb8\") "
Sep 16 14:59:41.753128 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.753053    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8f5f73e-2b75-4541-99c0-b4f572a41fb8" (UID: "f8f5f73e-2b75-4541-99c0-b4f572a41fb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:59:41.753316 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.753189    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8f5f73e-2b75-4541-99c0-b4f572a41fb8" (UID: "f8f5f73e-2b75-4541-99c0-b4f572a41fb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:59:41.753430 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.753402    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:59:41.753497 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.753424    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-home" (OuterVolumeSpecName: "home") pod "f8f5f73e-2b75-4541-99c0-b4f572a41fb8" (UID: "f8f5f73e-2b75-4541-99c0-b4f572a41fb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:59:41.753497 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.753435    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:59:41.755356 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.755325    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-dshm" (OuterVolumeSpecName: "dshm") pod "f8f5f73e-2b75-4541-99c0-b4f572a41fb8" (UID: "f8f5f73e-2b75-4541-99c0-b4f572a41fb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:59:41.755477 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.755412    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-kube-api-access-kvv4f" (OuterVolumeSpecName: "kube-api-access-kvv4f") pod "f8f5f73e-2b75-4541-99c0-b4f572a41fb8" (UID: "f8f5f73e-2b75-4541-99c0-b4f572a41fb8"). InnerVolumeSpecName "kube-api-access-kvv4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:59:41.755527 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.755452    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8f5f73e-2b75-4541-99c0-b4f572a41fb8" (UID: "f8f5f73e-2b75-4541-99c0-b4f572a41fb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:59:41.853920 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.853885    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:59:41.853920 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.853917    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:59:41.854126 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.853946    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvv4f\" (UniqueName: \"kubernetes.io/projected/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-kube-api-access-kvv4f\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:59:41.854126 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:41.853956    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f5f73e-2b75-4541-99c0-b4f572a41fb8-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 14:59:42.501301 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:42.501258    2558 generic.go:356] "Generic (PLEG): container finished" podID="f8f5f73e-2b75-4541-99c0-b4f572a41fb8" containerID="e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b" exitCode=0
Sep 16 14:59:42.501743 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:42.501337    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"
Sep 16 14:59:42.501743 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:42.501345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx" event={"ID":"f8f5f73e-2b75-4541-99c0-b4f572a41fb8","Type":"ContainerDied","Data":"e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b"}
Sep 16 14:59:42.501743 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:42.501390    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx" event={"ID":"f8f5f73e-2b75-4541-99c0-b4f572a41fb8","Type":"ContainerDied","Data":"f7063650cd854d1f403209cd1a0fff734287688548a9711d6b30ad7f643d9133"}
Sep 16 14:59:42.501743 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:42.501406    2558 scope.go:117] "RemoveContainer" containerID="e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b"
Sep 16 14:59:42.510377 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:42.510358    2558 scope.go:117] "RemoveContainer" containerID="e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b"
Sep 16 14:59:42.510622 ip-10-0-128-123 kubenswrapper[2558]: E0916 14:59:42.510603    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b\": container with ID starting with e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b not found: ID does not exist" containerID="e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b"
Sep 16 14:59:42.510678 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:42.510629    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b"} err="failed to get container status \"e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b\": rpc error: code = NotFound desc = could not find container \"e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b\": container with ID starting with e42b9be0199a53e75fa1b52814f593f4a894df738a29261c87578bef5a17490b not found: ID does not exist"
Sep 16 14:59:42.525621 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:42.525597    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"]
Sep 16 14:59:42.529529 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:42.529506    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-adb97937/e2e-lifecycle-model-adb97937-kserve-6bf788f86c-p8jrx"]
Sep 16 14:59:43.296244 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:43.296214    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8f5f73e-2b75-4541-99c0-b4f572a41fb8" path="/var/lib/kubelet/pods/f8f5f73e-2b75-4541-99c0-b4f572a41fb8/volumes"
Sep 16 14:59:48.916556 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:48.916489    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:48.917063 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:48.916580    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:48.928983 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:48.928962    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:49.552874 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:49.552846    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 14:59:57.927010 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:57.926972    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"]
Sep 16 14:59:57.927574 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:57.927559    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f5f73e-2b75-4541-99c0-b4f572a41fb8" containerName="main"
Sep 16 14:59:57.927618 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:57.927577    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f5f73e-2b75-4541-99c0-b4f572a41fb8" containerName="main"
Sep 16 14:59:57.927705 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:57.927695    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f5f73e-2b75-4541-99c0-b4f572a41fb8" containerName="main"
Sep 16 14:59:57.931025 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:57.931010    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:57.934707 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:57.934687    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e33b3f\"/\"distinct-2-w4-e33b3f-kserve-self-signed-certs\""
Sep 16 14:59:57.943138 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:57.943116    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"]
Sep 16 14:59:58.018970 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.018935    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d2vw\" (UniqueName: \"kubernetes.io/projected/8995ed84-0e3d-4421-b153-1fe3b487c85b-kube-api-access-2d2vw\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.019129 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.019001    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-dshm\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.019129 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.019044    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-home\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.019129 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.019064    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-tmp-dir\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.019129 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.019106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-model-cache\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.019293 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.019153    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8995ed84-0e3d-4421-b153-1fe3b487c85b-tls-certs\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.120535 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.120496    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-home\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.120535 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.120533    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-tmp-dir\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.120784 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.120554    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-model-cache\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.120784 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.120612    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8995ed84-0e3d-4421-b153-1fe3b487c85b-tls-certs\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.120784 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.120666    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d2vw\" (UniqueName: \"kubernetes.io/projected/8995ed84-0e3d-4421-b153-1fe3b487c85b-kube-api-access-2d2vw\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.120954 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.120852    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-dshm\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.121011 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.120954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-home\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.121011 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.120998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-tmp-dir\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.121090 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.121050    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-model-cache\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.122948 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.122924    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-dshm\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.123205 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.123178    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8995ed84-0e3d-4421-b153-1fe3b487c85b-tls-certs\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.129303 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.129276    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d2vw\" (UniqueName: \"kubernetes.io/projected/8995ed84-0e3d-4421-b153-1fe3b487c85b-kube-api-access-2d2vw\") pod \"distinct-2-w4-e33b3f-kserve-5689846856-dxgh5\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") " pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.240644 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.240560    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 14:59:58.517578 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.517553    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"]
Sep 16 14:59:58.520034 ip-10-0-128-123 kubenswrapper[2558]: W0916 14:59:58.520006    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8995ed84_0e3d_4421_b153_1fe3b487c85b.slice/crio-1f25f6ec9e654997824c3fc7549151b6e37b814234c211b4b1d112fcfac7de04 WatchSource:0}: Error finding container 1f25f6ec9e654997824c3fc7549151b6e37b814234c211b4b1d112fcfac7de04: Status 404 returned error can't find the container with id 1f25f6ec9e654997824c3fc7549151b6e37b814234c211b4b1d112fcfac7de04
Sep 16 14:59:58.588373 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.588347    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5" event={"ID":"8995ed84-0e3d-4421-b153-1fe3b487c85b","Type":"ContainerStarted","Data":"f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b"}
Sep 16 14:59:58.588504 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.588388    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5" event={"ID":"8995ed84-0e3d-4421-b153-1fe3b487c85b","Type":"ContainerStarted","Data":"1f25f6ec9e654997824c3fc7549151b6e37b814234c211b4b1d112fcfac7de04"}
Sep 16 14:59:58.612403 ip-10-0-128-123 kubenswrapper[2558]: I0916 14:59:58.612354    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5" podStartSLOduration=1.612338 podStartE2EDuration="1.612338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:59:58.60823881 +0000 UTC m=+1475.762317096" watchObservedRunningTime="2026-09-16 14:59:58.612338 +0000 UTC m=+1475.766416286"
Sep 16 15:00:00.188527 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.188482    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5"]
Sep 16 15:00:00.192270 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.192243    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5"
Sep 16 15:00:00.193757 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.193732    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td"]
Sep 16 15:00:00.195869 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.195831    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:00:00.201233 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.199306    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j"]
Sep 16 15:00:00.202417 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.201942    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td"
Sep 16 15:00:00.204650 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.204629    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs"]
Sep 16 15:00:00.204888 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.204844    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j"
Sep 16 15:00:00.206650 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.206629    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5"
Sep 16 15:00:00.215658 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.215620    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5"]
Sep 16 15:00:00.215893 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.215868    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td"]
Sep 16 15:00:00.216202 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.216154    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs"
Sep 16 15:00:00.219147 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.219129    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j"]
Sep 16 15:00:00.220941 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.220922    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td"
Sep 16 15:00:00.223722 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.223528    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs"]
Sep 16 15:00:00.224793 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.224773    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm"]
Sep 16 15:00:00.229022 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.229005    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm"
Sep 16 15:00:00.248833 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.248767    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm"]
Sep 16 15:00:00.263542 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.263518    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j"
Sep 16 15:00:00.275470 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.275445    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs"
Sep 16 15:00:00.326869 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.326828    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm"
Sep 16 15:00:00.516145 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.516027    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5"]
Sep 16 15:00:00.518745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.518723    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td"]
Sep 16 15:00:00.520009 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:00:00.519983    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cde76d_cc6e_4616_86c1_f7d8664f2e0c.slice/crio-63e5b9662036fbfb01450057a56ba5c9169e0a5d2951f290b1d6bc10fe978e2a WatchSource:0}: Error finding container 63e5b9662036fbfb01450057a56ba5c9169e0a5d2951f290b1d6bc10fe978e2a: Status 404 returned error can't find the container with id 63e5b9662036fbfb01450057a56ba5c9169e0a5d2951f290b1d6bc10fe978e2a
Sep 16 15:00:00.520626 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:00:00.520600    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d397300_5b86_441a_a540_f7f7051a4251.slice/crio-1043fb7aa0b521c5ee3520622b0a6b573a1906e5bb5fb1635c3a2401d0887af9 WatchSource:0}: Error finding container 1043fb7aa0b521c5ee3520622b0a6b573a1906e5bb5fb1635c3a2401d0887af9: Status 404 returned error can't find the container with id 1043fb7aa0b521c5ee3520622b0a6b573a1906e5bb5fb1635c3a2401d0887af9
Sep 16 15:00:00.582767 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.582747    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j"]
Sep 16 15:00:00.584577 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:00:00.584552    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda940beb6_1ae2_4774_ac89_33605adaa06d.slice/crio-84f2e42c01d860506f1516c3da9dbceae4be5218995405b785716841559d50a2 WatchSource:0}: Error finding container 84f2e42c01d860506f1516c3da9dbceae4be5218995405b785716841559d50a2: Status 404 returned error can't find the container with id 84f2e42c01d860506f1516c3da9dbceae4be5218995405b785716841559d50a2
Sep 16 15:00:00.596897 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.596875    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs"]
Sep 16 15:00:00.599617 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:00:00.599597    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3ed8c1_2562_40bd_a715_48e6b73f2976.slice/crio-c5212656a62bdcad2becdfd78c8849933142dc6ebd5320621d3e2776a4292668 WatchSource:0}: Error finding container c5212656a62bdcad2becdfd78c8849933142dc6ebd5320621d3e2776a4292668: Status 404 returned error can't find the container with id c5212656a62bdcad2becdfd78c8849933142dc6ebd5320621d3e2776a4292668
Sep 16 15:00:00.601606 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.601582    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5" event={"ID":"c3cde76d-cc6e-4616-86c1-f7d8664f2e0c","Type":"ContainerStarted","Data":"63e5b9662036fbfb01450057a56ba5c9169e0a5d2951f290b1d6bc10fe978e2a"}
Sep 16 15:00:00.602994 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.602971    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j" event={"ID":"a940beb6-1ae2-4774-ac89-33605adaa06d","Type":"ContainerStarted","Data":"84f2e42c01d860506f1516c3da9dbceae4be5218995405b785716841559d50a2"}
Sep 16 15:00:00.604246 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.604220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td" event={"ID":"8d397300-5b86-441a-a540-f7f7051a4251","Type":"ContainerStarted","Data":"1043fb7aa0b521c5ee3520622b0a6b573a1906e5bb5fb1635c3a2401d0887af9"}
Sep 16 15:00:00.631370 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:00.631346    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm"]
Sep 16 15:00:00.633382 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:00:00.633359    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13508f81_3522_4230_9177_a6c78ff2f773.slice/crio-cbe672d97d98b5caea8b212f710c480bf1075ad8023756930c18fad0ea54cf2b WatchSource:0}: Error finding container cbe672d97d98b5caea8b212f710c480bf1075ad8023756930c18fad0ea54cf2b: Status 404 returned error can't find the container with id cbe672d97d98b5caea8b212f710c480bf1075ad8023756930c18fad0ea54cf2b
Sep 16 15:00:01.612135 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:01.612099    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs" event={"ID":"5b3ed8c1-2562-40bd-a715-48e6b73f2976","Type":"ContainerStarted","Data":"0cedf449322f52f089005d9f0b9d1c72164b7e025848f5d13e043d93ca5374fc"}
Sep 16 15:00:01.612579 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:01.612158    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs" event={"ID":"5b3ed8c1-2562-40bd-a715-48e6b73f2976","Type":"ContainerStarted","Data":"c5212656a62bdcad2becdfd78c8849933142dc6ebd5320621d3e2776a4292668"}
Sep 16 15:00:01.613616 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:01.613589    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5" event={"ID":"c3cde76d-cc6e-4616-86c1-f7d8664f2e0c","Type":"ContainerStarted","Data":"5a98de290e1475f221d7edd19de1056a6ce41b99d412fd39c69e9a129e3f877c"}
Sep 16 15:00:01.615225 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:01.615195    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j" event={"ID":"a940beb6-1ae2-4774-ac89-33605adaa06d","Type":"ContainerStarted","Data":"2c8303af4f4864e17342f27229bbd9a2776d46de859d502e206db974800f48df"}
Sep 16 15:00:01.616797 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:01.616775    2558 generic.go:356] "Generic (PLEG): container finished" podID="8d397300-5b86-441a-a540-f7f7051a4251" containerID="4f5675d869daa4b0a03aaf0c39d5c7f9b049985ab23a64b42e7534091154fb2a" exitCode=0
Sep 16 15:00:01.616917 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:01.616843    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td" event={"ID":"8d397300-5b86-441a-a540-f7f7051a4251","Type":"ContainerDied","Data":"4f5675d869daa4b0a03aaf0c39d5c7f9b049985ab23a64b42e7534091154fb2a"}
Sep 16 15:00:01.618238 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:01.618220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm" event={"ID":"13508f81-3522-4230-9177-a6c78ff2f773","Type":"ContainerStarted","Data":"33ed67c29d6860eb3538d6142d186a29affee2dcefecce98824fd2203ab3b415"}
Sep 16 15:00:01.618318 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:01.618253    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm" event={"ID":"13508f81-3522-4230-9177-a6c78ff2f773","Type":"ContainerStarted","Data":"cbe672d97d98b5caea8b212f710c480bf1075ad8023756930c18fad0ea54cf2b"}
Sep 16 15:00:01.631641 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:01.631605    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs" podStartSLOduration=1.631591982 podStartE2EDuration="1.631591982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:00:01.629161227 +0000 UTC m=+1478.783239515" watchObservedRunningTime="2026-09-16 15:00:01.631591982 +0000 UTC m=+1478.785670269"
Sep 16 15:00:01.647115 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:01.647079    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5" podStartSLOduration=1.647069889 podStartE2EDuration="1.647069889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:00:01.646629718 +0000 UTC m=+1478.800708004" watchObservedRunningTime="2026-09-16 15:00:01.647069889 +0000 UTC m=+1478.801148238"
Sep 16 15:00:01.680190 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:01.680137    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm" podStartSLOduration=1.680123782 podStartE2EDuration="1.680123782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:00:01.679380397 +0000 UTC m=+1478.833458684" watchObservedRunningTime="2026-09-16 15:00:01.680123782 +0000 UTC m=+1478.834202065"
Sep 16 15:00:02.625238 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:02.625207    2558 generic.go:356] "Generic (PLEG): container finished" podID="c3cde76d-cc6e-4616-86c1-f7d8664f2e0c" containerID="5a98de290e1475f221d7edd19de1056a6ce41b99d412fd39c69e9a129e3f877c" exitCode=0
Sep 16 15:00:02.625685 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:02.625281    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5" event={"ID":"c3cde76d-cc6e-4616-86c1-f7d8664f2e0c","Type":"ContainerDied","Data":"5a98de290e1475f221d7edd19de1056a6ce41b99d412fd39c69e9a129e3f877c"}
Sep 16 15:00:02.626818 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:02.626787    2558 generic.go:356] "Generic (PLEG): container finished" podID="a940beb6-1ae2-4774-ac89-33605adaa06d" containerID="2c8303af4f4864e17342f27229bbd9a2776d46de859d502e206db974800f48df" exitCode=0
Sep 16 15:00:02.626918 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:02.626871    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j" event={"ID":"a940beb6-1ae2-4774-ac89-33605adaa06d","Type":"ContainerDied","Data":"2c8303af4f4864e17342f27229bbd9a2776d46de859d502e206db974800f48df"}
Sep 16 15:00:02.628298 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:02.628277    2558 generic.go:356] "Generic (PLEG): container finished" podID="13508f81-3522-4230-9177-a6c78ff2f773" containerID="33ed67c29d6860eb3538d6142d186a29affee2dcefecce98824fd2203ab3b415" exitCode=0
Sep 16 15:00:02.628378 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:02.628354    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm" event={"ID":"13508f81-3522-4230-9177-a6c78ff2f773","Type":"ContainerDied","Data":"33ed67c29d6860eb3538d6142d186a29affee2dcefecce98824fd2203ab3b415"}
Sep 16 15:00:02.629825 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:02.629809    2558 generic.go:356] "Generic (PLEG): container finished" podID="5b3ed8c1-2562-40bd-a715-48e6b73f2976" containerID="0cedf449322f52f089005d9f0b9d1c72164b7e025848f5d13e043d93ca5374fc" exitCode=0
Sep 16 15:00:02.629900 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:02.629830    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs" event={"ID":"5b3ed8c1-2562-40bd-a715-48e6b73f2976","Type":"ContainerDied","Data":"0cedf449322f52f089005d9f0b9d1c72164b7e025848f5d13e043d93ca5374fc"}
Sep 16 15:00:02.644093 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:02.644049    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j" podStartSLOduration=2.644034537 podStartE2EDuration="2.644034537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:00:01.702748269 +0000 UTC m=+1478.856826557" watchObservedRunningTime="2026-09-16 15:00:02.644034537 +0000 UTC m=+1479.798112824"
Sep 16 15:00:02.765732 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:02.765709    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td"
Sep 16 15:00:03.637967 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:03.637883    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td"
Sep 16 15:00:03.637967 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:03.637928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td" event={"ID":"8d397300-5b86-441a-a540-f7f7051a4251","Type":"ContainerDied","Data":"1043fb7aa0b521c5ee3520622b0a6b573a1906e5bb5fb1635c3a2401d0887af9"}
Sep 16 15:00:03.638438 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:03.637977    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1043fb7aa0b521c5ee3520622b0a6b573a1906e5bb5fb1635c3a2401d0887af9"
Sep 16 15:00:03.813126 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:03.812864    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs"
Sep 16 15:00:03.817372 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:03.817352    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm"
Sep 16 15:00:03.889727 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:03.889660    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5"
Sep 16 15:00:03.894268 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:03.894244    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j"
Sep 16 15:00:04.645003 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.644976    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5"
Sep 16 15:00:04.645437 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.644976    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826180-sr7t5" event={"ID":"c3cde76d-cc6e-4616-86c1-f7d8664f2e0c","Type":"ContainerDied","Data":"63e5b9662036fbfb01450057a56ba5c9169e0a5d2951f290b1d6bc10fe978e2a"}
Sep 16 15:00:04.645437 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.645091    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63e5b9662036fbfb01450057a56ba5c9169e0a5d2951f290b1d6bc10fe978e2a"
Sep 16 15:00:04.646334 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.646311    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j"
Sep 16 15:00:04.646449 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.646332    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j" event={"ID":"a940beb6-1ae2-4774-ac89-33605adaa06d","Type":"ContainerDied","Data":"84f2e42c01d860506f1516c3da9dbceae4be5218995405b785716841559d50a2"}
Sep 16 15:00:04.646449 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.646375    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84f2e42c01d860506f1516c3da9dbceae4be5218995405b785716841559d50a2"
Sep 16 15:00:04.647628 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.647613    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm"
Sep 16 15:00:04.647744 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.647651    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm" event={"ID":"13508f81-3522-4230-9177-a6c78ff2f773","Type":"ContainerDied","Data":"cbe672d97d98b5caea8b212f710c480bf1075ad8023756930c18fad0ea54cf2b"}
Sep 16 15:00:04.647744 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.647687    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbe672d97d98b5caea8b212f710c480bf1075ad8023756930c18fad0ea54cf2b"
Sep 16 15:00:04.649102 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.649082    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs" event={"ID":"5b3ed8c1-2562-40bd-a715-48e6b73f2976","Type":"ContainerDied","Data":"c5212656a62bdcad2becdfd78c8849933142dc6ebd5320621d3e2776a4292668"}
Sep 16 15:00:04.649214 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.649129    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5212656a62bdcad2becdfd78c8849933142dc6ebd5320621d3e2776a4292668"
Sep 16 15:00:04.649214 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:04.649091    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs"
Sep 16 15:00:08.241121 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:08.241088    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 15:00:08.241636 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:08.241250    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 15:00:08.253968 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:08.253938    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 15:00:08.683663 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:08.683638    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 15:00:23.296766 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:23.296736    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 15:00:23.301112 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:23.301093    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 15:00:41.426524 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.426492    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"]
Sep 16 15:00:41.426935 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.426890    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3cde76d-cc6e-4616-86c1-f7d8664f2e0c" containerName="cleanup"
Sep 16 15:00:41.426935 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.426902    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3cde76d-cc6e-4616-86c1-f7d8664f2e0c" containerName="cleanup"
Sep 16 15:00:41.426935 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.426913    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a940beb6-1ae2-4774-ac89-33605adaa06d" containerName="cleanup"
Sep 16 15:00:41.426935 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.426918    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a940beb6-1ae2-4774-ac89-33605adaa06d" containerName="cleanup"
Sep 16 15:00:41.426935 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.426935    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13508f81-3522-4230-9177-a6c78ff2f773" containerName="cleanup"
Sep 16 15:00:41.427120 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.426941    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="13508f81-3522-4230-9177-a6c78ff2f773" containerName="cleanup"
Sep 16 15:00:41.427120 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.426948    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d397300-5b86-441a-a540-f7f7051a4251" containerName="cleanup"
Sep 16 15:00:41.427120 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.426953    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d397300-5b86-441a-a540-f7f7051a4251" containerName="cleanup"
Sep 16 15:00:41.427120 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.426961    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b3ed8c1-2562-40bd-a715-48e6b73f2976" containerName="cleanup"
Sep 16 15:00:41.427120 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.426966    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b3ed8c1-2562-40bd-a715-48e6b73f2976" containerName="cleanup"
Sep 16 15:00:41.427120 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.427026    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a940beb6-1ae2-4774-ac89-33605adaa06d" containerName="cleanup"
Sep 16 15:00:41.427120 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.427034    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d397300-5b86-441a-a540-f7f7051a4251" containerName="cleanup"
Sep 16 15:00:41.427120 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.427042    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3cde76d-cc6e-4616-86c1-f7d8664f2e0c" containerName="cleanup"
Sep 16 15:00:41.427120 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.427048    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="13508f81-3522-4230-9177-a6c78ff2f773" containerName="cleanup"
Sep 16 15:00:41.427120 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.427053    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b3ed8c1-2562-40bd-a715-48e6b73f2976" containerName="cleanup"
Sep 16 15:00:41.431497 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.431479    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.434434 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.434416    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-780e10\"/\"embedding-w0-780e10-kserve-self-signed-certs\""
Sep 16 15:00:41.440196 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.440174    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"]
Sep 16 15:00:41.556952 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.556912    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tls-certs\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.557130 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.556972    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8q5\" (UniqueName: \"kubernetes.io/projected/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-kube-api-access-hr8q5\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.557130 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.557028    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-home\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.557130 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.557076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-model-cache\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.557130 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.557114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tmp-dir\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.557323 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.557198    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-dshm\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.658720 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.658684    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8q5\" (UniqueName: \"kubernetes.io/projected/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-kube-api-access-hr8q5\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.658915 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.658731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-home\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.658915 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.658760    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-model-cache\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.658915 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.658796    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tmp-dir\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.658915 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.658828    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-dshm\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.659136 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.658917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tls-certs\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.659263 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.659237    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-home\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.659263 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.659256    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-model-cache\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.659373 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.659306    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tmp-dir\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.661274 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.661257    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-dshm\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.661567 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.661545    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tls-certs\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.668127 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.668107    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8q5\" (UniqueName: \"kubernetes.io/projected/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-kube-api-access-hr8q5\") pod \"embedding-w0-780e10-kserve-6585f979bc-v4hvw\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") " pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:41.742277 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:41.742197    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:42.019750 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:42.019724    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"]
Sep 16 15:00:42.021808 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:00:42.021780    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fa2dc9d_2d80_42d4_844b_ce06deb096ed.slice/crio-d5f78a8478d61484186eff8893aff831be7fa3d3df5cd764fc5478ce8ceff00e WatchSource:0}: Error finding container d5f78a8478d61484186eff8893aff831be7fa3d3df5cd764fc5478ce8ceff00e: Status 404 returned error can't find the container with id d5f78a8478d61484186eff8893aff831be7fa3d3df5cd764fc5478ce8ceff00e
Sep 16 15:00:42.868449 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:42.868417    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw" event={"ID":"8fa2dc9d-2d80-42d4-844b-ce06deb096ed","Type":"ContainerStarted","Data":"e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0"}
Sep 16 15:00:42.868854 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:42.868459    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw" event={"ID":"8fa2dc9d-2d80-42d4-844b-ce06deb096ed","Type":"ContainerStarted","Data":"d5f78a8478d61484186eff8893aff831be7fa3d3df5cd764fc5478ce8ceff00e"}
Sep 16 15:00:42.890806 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:42.890757    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw" podStartSLOduration=1.89074222 podStartE2EDuration="1.89074222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:00:42.888418028 +0000 UTC m=+1520.042496316" watchObservedRunningTime="2026-09-16 15:00:42.89074222 +0000 UTC m=+1520.044820507"
Sep 16 15:00:51.743345 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:51.743254    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:51.743345 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:51.743318    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:51.755922 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:51.755897    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:00:51.929650 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:00:51.929622    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:01:23.498891 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:23.498858    2558 scope.go:117] "RemoveContainer" containerID="b855cc629cabebdd962e71dd0efaeb06b09b54467a1d39f46c7b5f446583acca"
Sep 16 15:01:23.507259 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:23.507232    2558 scope.go:117] "RemoveContainer" containerID="b2600d940d560b3c13f41cbc865bd4af06a708864e113cab504efe0af7290741"
Sep 16 15:01:47.496318 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:47.496280    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"]
Sep 16 15:01:47.499627 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:47.499608    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"
Sep 16 15:01:47.505072 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:47.505047    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:01:47.530625 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:47.530598    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"]
Sep 16 15:01:47.624787 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:47.624751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c7a55-04bc-4a24-9043-688950e45fe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6\" (UID: \"ef9c7a55-04bc-4a24-9043-688950e45fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"
Sep 16 15:01:47.726107 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:47.726066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c7a55-04bc-4a24-9043-688950e45fe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6\" (UID: \"ef9c7a55-04bc-4a24-9043-688950e45fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"
Sep 16 15:01:47.726785 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:47.726760    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c7a55-04bc-4a24-9043-688950e45fe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6\" (UID: \"ef9c7a55-04bc-4a24-9043-688950e45fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"
Sep 16 15:01:47.810854 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:47.810821    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"
Sep 16 15:01:48.096041 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:48.096015    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"]
Sep 16 15:01:48.097946 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:01:48.097920    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9c7a55_04bc_4a24_9043_688950e45fe6.slice/crio-b65dbec3443d4090d756860fe79255f7d2f4b54d4d1468fa93dc1d8e97bcbda7 WatchSource:0}: Error finding container b65dbec3443d4090d756860fe79255f7d2f4b54d4d1468fa93dc1d8e97bcbda7: Status 404 returned error can't find the container with id b65dbec3443d4090d756860fe79255f7d2f4b54d4d1468fa93dc1d8e97bcbda7
Sep 16 15:01:48.237611 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:48.237578    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6" event={"ID":"ef9c7a55-04bc-4a24-9043-688950e45fe6","Type":"ContainerStarted","Data":"b65dbec3443d4090d756860fe79255f7d2f4b54d4d1468fa93dc1d8e97bcbda7"}
Sep 16 15:01:49.246589 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:49.246555    2558 generic.go:356] "Generic (PLEG): container finished" podID="ef9c7a55-04bc-4a24-9043-688950e45fe6" containerID="0874aec5051d7cb2302c8853a13b66989cfa34dad05edece602d63d5fd109462" exitCode=0
Sep 16 15:01:49.246982 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:49.246632    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6" event={"ID":"ef9c7a55-04bc-4a24-9043-688950e45fe6","Type":"ContainerDied","Data":"0874aec5051d7cb2302c8853a13b66989cfa34dad05edece602d63d5fd109462"}
Sep 16 15:01:50.397444 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:50.397414    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"
Sep 16 15:01:50.554233 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:50.554204    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c7a55-04bc-4a24-9043-688950e45fe6-maas-api-service-ca\") pod \"ef9c7a55-04bc-4a24-9043-688950e45fe6\" (UID: \"ef9c7a55-04bc-4a24-9043-688950e45fe6\") "
Sep 16 15:01:50.554582 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:50.554558    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9c7a55-04bc-4a24-9043-688950e45fe6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef9c7a55-04bc-4a24-9043-688950e45fe6" (UID: "ef9c7a55-04bc-4a24-9043-688950e45fe6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:01:50.654960 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:50.654918    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9c7a55-04bc-4a24-9043-688950e45fe6-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:01:51.258497 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:51.258472    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"
Sep 16 15:01:51.258686 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:51.258475    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6" event={"ID":"ef9c7a55-04bc-4a24-9043-688950e45fe6","Type":"ContainerDied","Data":"b65dbec3443d4090d756860fe79255f7d2f4b54d4d1468fa93dc1d8e97bcbda7"}
Sep 16 15:01:51.258686 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:51.258538    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b65dbec3443d4090d756860fe79255f7d2f4b54d4d1468fa93dc1d8e97bcbda7"
Sep 16 15:01:52.352521 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.352485    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"]
Sep 16 15:01:52.364151 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.364118    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"]
Sep 16 15:01:52.752448 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.752367    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"]
Sep 16 15:01:52.752809 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.752797    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef9c7a55-04bc-4a24-9043-688950e45fe6" containerName="revoke-keys"
Sep 16 15:01:52.752855 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.752811    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef9c7a55-04bc-4a24-9043-688950e45fe6" containerName="revoke-keys"
Sep 16 15:01:52.752892 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.752885    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef9c7a55-04bc-4a24-9043-688950e45fe6" containerName="revoke-keys"
Sep 16 15:01:52.756004 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.755984    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:01:52.759773 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.759752    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-079e6e-dockercfg-2qxgh\""
Sep 16 15:01:52.759872 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.759779    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-079e6e\""
Sep 16 15:01:52.765729 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.765706    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"]
Sep 16 15:01:52.834696 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.834666    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"]
Sep 16 15:01:52.838455 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.838430    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:01:52.852492 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.852462    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"]
Sep 16 15:01:52.878354 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.878318    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gvpp\" (UniqueName: \"kubernetes.io/projected/231b64dd-d14a-488f-ae45-2334a423364a-kube-api-access-6gvpp\") pod \"payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb\" (UID: \"231b64dd-d14a-488f-ae45-2334a423364a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:01:52.878547 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.878407    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231b64dd-d14a-488f-ae45-2334a423364a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb\" (UID: \"231b64dd-d14a-488f-ae45-2334a423364a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:01:52.979732 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.979695    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6q5x\" (UniqueName: \"kubernetes.io/projected/3046c826-b1e4-4486-b46b-cc72ef8a51b1-kube-api-access-s6q5x\") pod \"payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx\" (UID: \"3046c826-b1e4-4486-b46b-cc72ef8a51b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:01:52.979908 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.979830    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gvpp\" (UniqueName: \"kubernetes.io/projected/231b64dd-d14a-488f-ae45-2334a423364a-kube-api-access-6gvpp\") pod \"payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb\" (UID: \"231b64dd-d14a-488f-ae45-2334a423364a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:01:52.979908 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.979892    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3046c826-b1e4-4486-b46b-cc72ef8a51b1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx\" (UID: \"3046c826-b1e4-4486-b46b-cc72ef8a51b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:01:52.980016 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.979943    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231b64dd-d14a-488f-ae45-2334a423364a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb\" (UID: \"231b64dd-d14a-488f-ae45-2334a423364a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:01:52.980645 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.980624    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231b64dd-d14a-488f-ae45-2334a423364a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb\" (UID: \"231b64dd-d14a-488f-ae45-2334a423364a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:01:52.988498 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:52.988473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gvpp\" (UniqueName: \"kubernetes.io/projected/231b64dd-d14a-488f-ae45-2334a423364a-kube-api-access-6gvpp\") pod \"payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb\" (UID: \"231b64dd-d14a-488f-ae45-2334a423364a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:01:53.067652 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:53.067625    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:01:53.081316 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:53.081290    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6q5x\" (UniqueName: \"kubernetes.io/projected/3046c826-b1e4-4486-b46b-cc72ef8a51b1-kube-api-access-s6q5x\") pod \"payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx\" (UID: \"3046c826-b1e4-4486-b46b-cc72ef8a51b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:01:53.081455 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:53.081398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3046c826-b1e4-4486-b46b-cc72ef8a51b1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx\" (UID: \"3046c826-b1e4-4486-b46b-cc72ef8a51b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:01:53.082255 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:53.082230    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3046c826-b1e4-4486-b46b-cc72ef8a51b1-plugins-config-volume\") pod \"payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx\" (UID: \"3046c826-b1e4-4486-b46b-cc72ef8a51b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:01:53.090109 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:53.090084    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6q5x\" (UniqueName: \"kubernetes.io/projected/3046c826-b1e4-4486-b46b-cc72ef8a51b1-kube-api-access-s6q5x\") pod \"payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx\" (UID: \"3046c826-b1e4-4486-b46b-cc72ef8a51b1\") " pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:01:53.148195 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:53.148146    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:01:53.344216 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:53.344186    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"]
Sep 16 15:01:53.345321 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:01:53.345292    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231b64dd_d14a_488f_ae45_2334a423364a.slice/crio-b3040e3f86852e67a62eaeaae726290027c4db939ea3df06e2832129bcdace9c WatchSource:0}: Error finding container b3040e3f86852e67a62eaeaae726290027c4db939ea3df06e2832129bcdace9c: Status 404 returned error can't find the container with id b3040e3f86852e67a62eaeaae726290027c4db939ea3df06e2832129bcdace9c
Sep 16 15:01:53.428672 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:53.428633    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"]
Sep 16 15:01:53.430737 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:01:53.430712    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3046c826_b1e4_4486_b46b_cc72ef8a51b1.slice/crio-0b6cd9187a87a7b7d51d7715166decce7adf57c8077276cb4f69606712c364fa WatchSource:0}: Error finding container 0b6cd9187a87a7b7d51d7715166decce7adf57c8077276cb4f69606712c364fa: Status 404 returned error can't find the container with id 0b6cd9187a87a7b7d51d7715166decce7adf57c8077276cb4f69606712c364fa
Sep 16 15:01:54.280961 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:54.280925    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb" event={"ID":"231b64dd-d14a-488f-ae45-2334a423364a","Type":"ContainerStarted","Data":"31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598"}
Sep 16 15:01:54.281193 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:54.280993    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb" event={"ID":"231b64dd-d14a-488f-ae45-2334a423364a","Type":"ContainerStarted","Data":"b3040e3f86852e67a62eaeaae726290027c4db939ea3df06e2832129bcdace9c"}
Sep 16 15:01:54.281193 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:54.281028    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:01:54.282802 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:54.282776    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx" event={"ID":"3046c826-b1e4-4486-b46b-cc72ef8a51b1","Type":"ContainerStarted","Data":"a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661"}
Sep 16 15:01:54.282941 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:54.282826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx" event={"ID":"3046c826-b1e4-4486-b46b-cc72ef8a51b1","Type":"ContainerStarted","Data":"0b6cd9187a87a7b7d51d7715166decce7adf57c8077276cb4f69606712c364fa"}
Sep 16 15:01:54.282941 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:54.282863    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:01:54.298889 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:54.298834    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb" podStartSLOduration=2.298819965 podStartE2EDuration="2.298819965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:01:54.298149858 +0000 UTC m=+1591.452228147" watchObservedRunningTime="2026-09-16 15:01:54.298819965 +0000 UTC m=+1591.452898252"
Sep 16 15:01:54.317523 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:01:54.317452    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx" podStartSLOduration=2.31741476 podStartE2EDuration="2.31741476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:01:54.314529945 +0000 UTC m=+1591.468608233" watchObservedRunningTime="2026-09-16 15:01:54.31741476 +0000 UTC m=+1591.471493048"
Sep 16 15:02:03.351882 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.351851    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs"]
Sep 16 15:02:03.356940 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.356914    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-780e10-29826180-8bmcs"]
Sep 16 15:02:03.381462 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.381434    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"]
Sep 16 15:02:03.381671 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.381646    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr" podUID="7456814a-59b2-42ca-a415-9ffd7b975e91" containerName="payload-processing" containerID="cri-o://c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a" gracePeriod=30
Sep 16 15:02:03.394773 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.393604    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"]
Sep 16 15:02:03.394773 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.393957    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d" podUID="07cfd061-a487-4331-9114-dfa8e3cfc1a1" containerName="payload-pre-processing" containerID="cri-o://a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812" gracePeriod=30
Sep 16 15:02:03.571404 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.570949    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"]
Sep 16 15:02:03.573272 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.573210    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-780e10-dhpw6"]
Sep 16 15:02:03.686515 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.686495    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 15:02:03.689773 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.689753    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 15:02:03.787370 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.787341    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07cfd061-a487-4331-9114-dfa8e3cfc1a1-plugins-config-volume\") pod \"07cfd061-a487-4331-9114-dfa8e3cfc1a1\" (UID: \"07cfd061-a487-4331-9114-dfa8e3cfc1a1\") "
Sep 16 15:02:03.787542 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.787380    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7456814a-59b2-42ca-a415-9ffd7b975e91-plugins-config-volume\") pod \"7456814a-59b2-42ca-a415-9ffd7b975e91\" (UID: \"7456814a-59b2-42ca-a415-9ffd7b975e91\") "
Sep 16 15:02:03.787542 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.787419    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhgrm\" (UniqueName: \"kubernetes.io/projected/7456814a-59b2-42ca-a415-9ffd7b975e91-kube-api-access-bhgrm\") pod \"7456814a-59b2-42ca-a415-9ffd7b975e91\" (UID: \"7456814a-59b2-42ca-a415-9ffd7b975e91\") "
Sep 16 15:02:03.787681 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.787592    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ztdf\" (UniqueName: \"kubernetes.io/projected/07cfd061-a487-4331-9114-dfa8e3cfc1a1-kube-api-access-5ztdf\") pod \"07cfd061-a487-4331-9114-dfa8e3cfc1a1\" (UID: \"07cfd061-a487-4331-9114-dfa8e3cfc1a1\") "
Sep 16 15:02:03.787812 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.787788    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07cfd061-a487-4331-9114-dfa8e3cfc1a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07cfd061-a487-4331-9114-dfa8e3cfc1a1" (UID: "07cfd061-a487-4331-9114-dfa8e3cfc1a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:02:03.787812 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.787796    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7456814a-59b2-42ca-a415-9ffd7b975e91-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7456814a-59b2-42ca-a415-9ffd7b975e91" (UID: "7456814a-59b2-42ca-a415-9ffd7b975e91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:02:03.788013 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.787992    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07cfd061-a487-4331-9114-dfa8e3cfc1a1-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:03.788062 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.788020    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7456814a-59b2-42ca-a415-9ffd7b975e91-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:03.790226 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.790199    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7456814a-59b2-42ca-a415-9ffd7b975e91-kube-api-access-bhgrm" (OuterVolumeSpecName: "kube-api-access-bhgrm") pod "7456814a-59b2-42ca-a415-9ffd7b975e91" (UID: "7456814a-59b2-42ca-a415-9ffd7b975e91"). InnerVolumeSpecName "kube-api-access-bhgrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:02:03.790316 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.790265    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07cfd061-a487-4331-9114-dfa8e3cfc1a1-kube-api-access-5ztdf" (OuterVolumeSpecName: "kube-api-access-5ztdf") pod "07cfd061-a487-4331-9114-dfa8e3cfc1a1" (UID: "07cfd061-a487-4331-9114-dfa8e3cfc1a1"). InnerVolumeSpecName "kube-api-access-5ztdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:02:03.888765 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.888728    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ztdf\" (UniqueName: \"kubernetes.io/projected/07cfd061-a487-4331-9114-dfa8e3cfc1a1-kube-api-access-5ztdf\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:03.888765 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:03.888760    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhgrm\" (UniqueName: \"kubernetes.io/projected/7456814a-59b2-42ca-a415-9ffd7b975e91-kube-api-access-bhgrm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:04.342701 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.342665    2558 generic.go:356] "Generic (PLEG): container finished" podID="07cfd061-a487-4331-9114-dfa8e3cfc1a1" containerID="a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812" exitCode=0
Sep 16 15:02:04.342898 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.342712    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"
Sep 16 15:02:04.342898 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.342752    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d" event={"ID":"07cfd061-a487-4331-9114-dfa8e3cfc1a1","Type":"ContainerDied","Data":"a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812"}
Sep 16 15:02:04.342898 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.342809    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d" event={"ID":"07cfd061-a487-4331-9114-dfa8e3cfc1a1","Type":"ContainerDied","Data":"3a954a853e67d39412dafe161e844aa2c107867a658245c424c864b9019052e8"}
Sep 16 15:02:04.342898 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.342841    2558 scope.go:117] "RemoveContainer" containerID="a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812"
Sep 16 15:02:04.344407 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.344380    2558 generic.go:356] "Generic (PLEG): container finished" podID="7456814a-59b2-42ca-a415-9ffd7b975e91" containerID="c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a" exitCode=0
Sep 16 15:02:04.344527 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.344424    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"
Sep 16 15:02:04.344527 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.344466    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr" event={"ID":"7456814a-59b2-42ca-a415-9ffd7b975e91","Type":"ContainerDied","Data":"c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a"}
Sep 16 15:02:04.344527 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.344507    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr" event={"ID":"7456814a-59b2-42ca-a415-9ffd7b975e91","Type":"ContainerDied","Data":"e70fb80686b4e49ab8b4bd177cf1e07ddd3947a356ddffe4a784e2a53eed347f"}
Sep 16 15:02:04.353812 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.353796    2558 scope.go:117] "RemoveContainer" containerID="a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812"
Sep 16 15:02:04.354204 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:02:04.354185    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812\": container with ID starting with a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812 not found: ID does not exist" containerID="a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812"
Sep 16 15:02:04.354289 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.354213    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812"} err="failed to get container status \"a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812\": rpc error: code = NotFound desc = could not find container \"a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812\": container with ID starting with a1c6813b77e142a5d40dd008840c409a6111f9077b6d1c5c7e2130f45549c812 not found: ID does not exist"
Sep 16 15:02:04.354289 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.354228    2558 scope.go:117] "RemoveContainer" containerID="c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a"
Sep 16 15:02:04.363095 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.363073    2558 scope.go:117] "RemoveContainer" containerID="c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a"
Sep 16 15:02:04.363394 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:02:04.363373    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a\": container with ID starting with c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a not found: ID does not exist" containerID="c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a"
Sep 16 15:02:04.363461 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.363402    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a"} err="failed to get container status \"c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a\": rpc error: code = NotFound desc = could not find container \"c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a\": container with ID starting with c8a7a3384db3ce900d841c15e799958eee218362cf79b3bdcb17e7b9f5a7a40a not found: ID does not exist"
Sep 16 15:02:04.371026 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.371001    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"]
Sep 16 15:02:04.382824 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.382796    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-780e10-6448787554-2rk4d"]
Sep 16 15:02:04.444727 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.444695    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"]
Sep 16 15:02:04.460012 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:04.459972    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-780e10-86946c5ff7-mtdpr"]
Sep 16 15:02:05.296360 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:05.296330    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07cfd061-a487-4331-9114-dfa8e3cfc1a1" path="/var/lib/kubelet/pods/07cfd061-a487-4331-9114-dfa8e3cfc1a1/volumes"
Sep 16 15:02:05.296651 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:05.296639    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b3ed8c1-2562-40bd-a715-48e6b73f2976" path="/var/lib/kubelet/pods/5b3ed8c1-2562-40bd-a715-48e6b73f2976/volumes"
Sep 16 15:02:05.296922 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:05.296912    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7456814a-59b2-42ca-a415-9ffd7b975e91" path="/var/lib/kubelet/pods/7456814a-59b2-42ca-a415-9ffd7b975e91/volumes"
Sep 16 15:02:05.297221 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:05.297210    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef9c7a55-04bc-4a24-9043-688950e45fe6" path="/var/lib/kubelet/pods/ef9c7a55-04bc-4a24-9043-688950e45fe6/volumes"
Sep 16 15:02:05.297790 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:05.297775    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:02:05.297841 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:05.297826    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:02:07.372397 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.372354    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw" podUID="8fa2dc9d-2d80-42d4-844b-ce06deb096ed" containerName="main" containerID="cri-o://e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0" gracePeriod=45
Sep 16 15:02:07.384545 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.384516    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2" podUID="59040847-f4ee-4371-a882-10fc0a2d4661" containerName="main" containerID="cri-o://a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29" gracePeriod=45
Sep 16 15:02:07.641194 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.641155    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:02:07.670881 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.670848    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 15:02:07.726317 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726288    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-tmp-dir\") pod \"59040847-f4ee-4371-a882-10fc0a2d4661\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") "
Sep 16 15:02:07.726495 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726334    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-home\") pod \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") "
Sep 16 15:02:07.726495 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726364    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-dshm\") pod \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") "
Sep 16 15:02:07.726495 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726394    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-model-cache\") pod \"59040847-f4ee-4371-a882-10fc0a2d4661\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") "
Sep 16 15:02:07.726495 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726466    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-model-cache\") pod \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") "
Sep 16 15:02:07.726724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726502    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tls-certs\") pod \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") "
Sep 16 15:02:07.726724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726539    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-dshm\") pod \"59040847-f4ee-4371-a882-10fc0a2d4661\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") "
Sep 16 15:02:07.726724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726565    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tmp-dir\") pod \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") "
Sep 16 15:02:07.726724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726566    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59040847-f4ee-4371-a882-10fc0a2d4661" (UID: "59040847-f4ee-4371-a882-10fc0a2d4661"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:02:07.726724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726605    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-home\") pod \"59040847-f4ee-4371-a882-10fc0a2d4661\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") "
Sep 16 15:02:07.726724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726638    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr8q5\" (UniqueName: \"kubernetes.io/projected/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-kube-api-access-hr8q5\") pod \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\" (UID: \"8fa2dc9d-2d80-42d4-844b-ce06deb096ed\") "
Sep 16 15:02:07.726724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726662    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59040847-f4ee-4371-a882-10fc0a2d4661-tls-certs\") pod \"59040847-f4ee-4371-a882-10fc0a2d4661\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") "
Sep 16 15:02:07.726724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726688    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9flq\" (UniqueName: \"kubernetes.io/projected/59040847-f4ee-4371-a882-10fc0a2d4661-kube-api-access-q9flq\") pod \"59040847-f4ee-4371-a882-10fc0a2d4661\" (UID: \"59040847-f4ee-4371-a882-10fc0a2d4661\") "
Sep 16 15:02:07.727128 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726749    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-home" (OuterVolumeSpecName: "home") pod "8fa2dc9d-2d80-42d4-844b-ce06deb096ed" (UID: "8fa2dc9d-2d80-42d4-844b-ce06deb096ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:02:07.727128 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.726936    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-home" (OuterVolumeSpecName: "home") pod "59040847-f4ee-4371-a882-10fc0a2d4661" (UID: "59040847-f4ee-4371-a882-10fc0a2d4661"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:02:07.727332 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.727125    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:07.727332 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.727145    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:07.727332 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.727177    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:07.727488 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.727379    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fa2dc9d-2d80-42d4-844b-ce06deb096ed" (UID: "8fa2dc9d-2d80-42d4-844b-ce06deb096ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:02:07.727729 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.727615    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-model-cache" (OuterVolumeSpecName: "model-cache") pod "59040847-f4ee-4371-a882-10fc0a2d4661" (UID: "59040847-f4ee-4371-a882-10fc0a2d4661"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:02:07.727729 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.727692    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fa2dc9d-2d80-42d4-844b-ce06deb096ed" (UID: "8fa2dc9d-2d80-42d4-844b-ce06deb096ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:02:07.729966 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.729920    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-dshm" (OuterVolumeSpecName: "dshm") pod "8fa2dc9d-2d80-42d4-844b-ce06deb096ed" (UID: "8fa2dc9d-2d80-42d4-844b-ce06deb096ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:02:07.730461 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.730434    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59040847-f4ee-4371-a882-10fc0a2d4661-kube-api-access-q9flq" (OuterVolumeSpecName: "kube-api-access-q9flq") pod "59040847-f4ee-4371-a882-10fc0a2d4661" (UID: "59040847-f4ee-4371-a882-10fc0a2d4661"). InnerVolumeSpecName "kube-api-access-q9flq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:02:07.731790 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.731755    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fa2dc9d-2d80-42d4-844b-ce06deb096ed" (UID: "8fa2dc9d-2d80-42d4-844b-ce06deb096ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:02:07.731983 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.731945    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-dshm" (OuterVolumeSpecName: "dshm") pod "59040847-f4ee-4371-a882-10fc0a2d4661" (UID: "59040847-f4ee-4371-a882-10fc0a2d4661"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:02:07.731983 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.731968    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59040847-f4ee-4371-a882-10fc0a2d4661-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59040847-f4ee-4371-a882-10fc0a2d4661" (UID: "59040847-f4ee-4371-a882-10fc0a2d4661"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:02:07.732260 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.732237    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-kube-api-access-hr8q5" (OuterVolumeSpecName: "kube-api-access-hr8q5") pod "8fa2dc9d-2d80-42d4-844b-ce06deb096ed" (UID: "8fa2dc9d-2d80-42d4-844b-ce06deb096ed"). InnerVolumeSpecName "kube-api-access-hr8q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:02:07.828144 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.828114    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:07.828144 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.828140    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:07.828144 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.828150    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:07.828441 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.828161    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:07.828441 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.828186    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59040847-f4ee-4371-a882-10fc0a2d4661-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:07.828441 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.828193    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:07.828441 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.828202    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr8q5\" (UniqueName: \"kubernetes.io/projected/8fa2dc9d-2d80-42d4-844b-ce06deb096ed-kube-api-access-hr8q5\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:07.828441 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.828212    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59040847-f4ee-4371-a882-10fc0a2d4661-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:07.828441 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:07.828221    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9flq\" (UniqueName: \"kubernetes.io/projected/59040847-f4ee-4371-a882-10fc0a2d4661-kube-api-access-q9flq\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:08.372565 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.372532    2558 generic.go:356] "Generic (PLEG): container finished" podID="59040847-f4ee-4371-a882-10fc0a2d4661" containerID="a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29" exitCode=0
Sep 16 15:02:08.373024 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.372584    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"
Sep 16 15:02:08.373024 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.372626    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2" event={"ID":"59040847-f4ee-4371-a882-10fc0a2d4661","Type":"ContainerDied","Data":"a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29"}
Sep 16 15:02:08.373024 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.372683    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2" event={"ID":"59040847-f4ee-4371-a882-10fc0a2d4661","Type":"ContainerDied","Data":"a09e8dbcb52fbc28cefeccff9f34a574caa64b6859d5552e87d147f48e79baf8"}
Sep 16 15:02:08.373024 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.372705    2558 scope.go:117] "RemoveContainer" containerID="a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29"
Sep 16 15:02:08.374123 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.374092    2558 generic.go:356] "Generic (PLEG): container finished" podID="8fa2dc9d-2d80-42d4-844b-ce06deb096ed" containerID="e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0" exitCode=0
Sep 16 15:02:08.374205 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.374145    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"
Sep 16 15:02:08.374205 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.374149    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw" event={"ID":"8fa2dc9d-2d80-42d4-844b-ce06deb096ed","Type":"ContainerDied","Data":"e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0"}
Sep 16 15:02:08.374287 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.374214    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw" event={"ID":"8fa2dc9d-2d80-42d4-844b-ce06deb096ed","Type":"ContainerDied","Data":"d5f78a8478d61484186eff8893aff831be7fa3d3df5cd764fc5478ce8ceff00e"}
Sep 16 15:02:08.385948 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.385425    2558 scope.go:117] "RemoveContainer" containerID="a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29"
Sep 16 15:02:08.386673 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:02:08.386640    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29\": container with ID starting with a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29 not found: ID does not exist" containerID="a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29"
Sep 16 15:02:08.386839 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.386810    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29"} err="failed to get container status \"a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29\": rpc error: code = NotFound desc = could not find container \"a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29\": container with ID starting with a10479e5a9df7a5f73473297fc8ace4c62abb0608a88750b6536908da4995f29 not found: ID does not exist"
Sep 16 15:02:08.386951 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.386841    2558 scope.go:117] "RemoveContainer" containerID="e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0"
Sep 16 15:02:08.399915 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.399897    2558 scope.go:117] "RemoveContainer" containerID="e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0"
Sep 16 15:02:08.400344 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:02:08.400317    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0\": container with ID starting with e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0 not found: ID does not exist" containerID="e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0"
Sep 16 15:02:08.400462 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.400356    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0"} err="failed to get container status \"e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0\": rpc error: code = NotFound desc = could not find container \"e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0\": container with ID starting with e4b01757dd0c05c5930e1eeea13ebc074c9a32f188b6a5354280613576215eb0 not found: ID does not exist"
Sep 16 15:02:08.416949 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.416927    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"]
Sep 16 15:02:08.424434 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.424415    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-780e10/premium-w0-780e10-kserve-56cc7b6bf5-st7l2"]
Sep 16 15:02:08.438342 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.438321    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"]
Sep 16 15:02:08.444724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:08.444703    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-780e10/embedding-w0-780e10-kserve-6585f979bc-v4hvw"]
Sep 16 15:02:09.295637 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:09.295606    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59040847-f4ee-4371-a882-10fc0a2d4661" path="/var/lib/kubelet/pods/59040847-f4ee-4371-a882-10fc0a2d4661/volumes"
Sep 16 15:02:09.296012 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:09.295999    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa2dc9d-2d80-42d4-844b-ce06deb096ed" path="/var/lib/kubelet/pods/8fa2dc9d-2d80-42d4-844b-ce06deb096ed/volumes"
Sep 16 15:02:18.900042 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.899953    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"]
Sep 16 15:02:18.900607 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900585    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa2dc9d-2d80-42d4-844b-ce06deb096ed" containerName="main"
Sep 16 15:02:18.900607 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900605    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa2dc9d-2d80-42d4-844b-ce06deb096ed" containerName="main"
Sep 16 15:02:18.900733 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900625    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59040847-f4ee-4371-a882-10fc0a2d4661" containerName="main"
Sep 16 15:02:18.900733 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900633    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="59040847-f4ee-4371-a882-10fc0a2d4661" containerName="main"
Sep 16 15:02:18.900733 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900654    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07cfd061-a487-4331-9114-dfa8e3cfc1a1" containerName="payload-pre-processing"
Sep 16 15:02:18.900733 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900665    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="07cfd061-a487-4331-9114-dfa8e3cfc1a1" containerName="payload-pre-processing"
Sep 16 15:02:18.900733 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900674    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7456814a-59b2-42ca-a415-9ffd7b975e91" containerName="payload-processing"
Sep 16 15:02:18.900733 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900683    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7456814a-59b2-42ca-a415-9ffd7b975e91" containerName="payload-processing"
Sep 16 15:02:18.901041 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900790    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="59040847-f4ee-4371-a882-10fc0a2d4661" containerName="main"
Sep 16 15:02:18.901041 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900802    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="07cfd061-a487-4331-9114-dfa8e3cfc1a1" containerName="payload-pre-processing"
Sep 16 15:02:18.901041 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900814    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7456814a-59b2-42ca-a415-9ffd7b975e91" containerName="payload-processing"
Sep 16 15:02:18.901041 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.900827    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa2dc9d-2d80-42d4-844b-ce06deb096ed" containerName="main"
Sep 16 15:02:18.905667 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.905646    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:02:18.908371 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.908350    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-eecd90\""
Sep 16 15:02:18.918097 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.918072    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"]
Sep 16 15:02:18.930927 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.930904    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"]
Sep 16 15:02:18.934692 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.934674    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:02:18.937698 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.937677    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-eecd90-dockercfg-sdtt6\""
Sep 16 15:02:18.946932 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:18.946913    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"]
Sep 16 15:02:19.041451 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.041420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vxkw\" (UniqueName: \"kubernetes.io/projected/26a68ccc-21b8-41e4-80da-1ea60e72fd64-kube-api-access-9vxkw\") pod \"payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq\" (UID: \"26a68ccc-21b8-41e4-80da-1ea60e72fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:02:19.041451 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.041456    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5sgn\" (UniqueName: \"kubernetes.io/projected/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-kube-api-access-w5sgn\") pod \"payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9\" (UID: \"d66bdb3b-72f7-4cda-b56d-5991c516c2d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:02:19.041644 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.041559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a68ccc-21b8-41e4-80da-1ea60e72fd64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq\" (UID: \"26a68ccc-21b8-41e4-80da-1ea60e72fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:02:19.041644 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.041588    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9\" (UID: \"d66bdb3b-72f7-4cda-b56d-5991c516c2d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:02:19.142085 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.142048    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a68ccc-21b8-41e4-80da-1ea60e72fd64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq\" (UID: \"26a68ccc-21b8-41e4-80da-1ea60e72fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:02:19.142289 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.142102    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9\" (UID: \"d66bdb3b-72f7-4cda-b56d-5991c516c2d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:02:19.142289 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.142141    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vxkw\" (UniqueName: \"kubernetes.io/projected/26a68ccc-21b8-41e4-80da-1ea60e72fd64-kube-api-access-9vxkw\") pod \"payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq\" (UID: \"26a68ccc-21b8-41e4-80da-1ea60e72fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:02:19.142289 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.142196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5sgn\" (UniqueName: \"kubernetes.io/projected/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-kube-api-access-w5sgn\") pod \"payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9\" (UID: \"d66bdb3b-72f7-4cda-b56d-5991c516c2d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:02:19.142762 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.142735    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a68ccc-21b8-41e4-80da-1ea60e72fd64-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq\" (UID: \"26a68ccc-21b8-41e4-80da-1ea60e72fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:02:19.142802 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.142788    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-plugins-config-volume\") pod \"payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9\" (UID: \"d66bdb3b-72f7-4cda-b56d-5991c516c2d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:02:19.151636 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.151573    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vxkw\" (UniqueName: \"kubernetes.io/projected/26a68ccc-21b8-41e4-80da-1ea60e72fd64-kube-api-access-9vxkw\") pod \"payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq\" (UID: \"26a68ccc-21b8-41e4-80da-1ea60e72fd64\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:02:19.151636 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.151611    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5sgn\" (UniqueName: \"kubernetes.io/projected/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-kube-api-access-w5sgn\") pod \"payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9\" (UID: \"d66bdb3b-72f7-4cda-b56d-5991c516c2d2\") " pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:02:19.216572 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.216546    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:02:19.245111 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.245082    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:02:19.493942 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.493919    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"]
Sep 16 15:02:19.495448 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:02:19.495423    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a68ccc_21b8_41e4_80da_1ea60e72fd64.slice/crio-a952006af5676cbd6724c9c9d3e6c0206fb11cf47934130866840f6124544e1e WatchSource:0}: Error finding container a952006af5676cbd6724c9c9d3e6c0206fb11cf47934130866840f6124544e1e: Status 404 returned error can't find the container with id a952006af5676cbd6724c9c9d3e6c0206fb11cf47934130866840f6124544e1e
Sep 16 15:02:19.531521 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:19.531434    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"]
Sep 16 15:02:19.532672 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:02:19.532651    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66bdb3b_72f7_4cda_b56d_5991c516c2d2.slice/crio-46af1eb4ae03b9b82cdc20090747b3c8fd5d95821bb01876faebd48fe9fa20b4 WatchSource:0}: Error finding container 46af1eb4ae03b9b82cdc20090747b3c8fd5d95821bb01876faebd48fe9fa20b4: Status 404 returned error can't find the container with id 46af1eb4ae03b9b82cdc20090747b3c8fd5d95821bb01876faebd48fe9fa20b4
Sep 16 15:02:20.453251 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:20.453210    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq" event={"ID":"26a68ccc-21b8-41e4-80da-1ea60e72fd64","Type":"ContainerStarted","Data":"658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0"}
Sep 16 15:02:20.453717 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:20.453325    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq" event={"ID":"26a68ccc-21b8-41e4-80da-1ea60e72fd64","Type":"ContainerStarted","Data":"a952006af5676cbd6724c9c9d3e6c0206fb11cf47934130866840f6124544e1e"}
Sep 16 15:02:20.453717 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:20.453354    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:02:20.455076 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:20.455053    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9" event={"ID":"d66bdb3b-72f7-4cda-b56d-5991c516c2d2","Type":"ContainerStarted","Data":"5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b"}
Sep 16 15:02:20.455212 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:20.455089    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9" event={"ID":"d66bdb3b-72f7-4cda-b56d-5991c516c2d2","Type":"ContainerStarted","Data":"46af1eb4ae03b9b82cdc20090747b3c8fd5d95821bb01876faebd48fe9fa20b4"}
Sep 16 15:02:20.455212 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:20.455143    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:02:20.470442 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:20.470396    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq" podStartSLOduration=2.47037752 podStartE2EDuration="2.47037752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:02:20.469907994 +0000 UTC m=+1617.623986282" watchObservedRunningTime="2026-09-16 15:02:20.47037752 +0000 UTC m=+1617.624455791"
Sep 16 15:02:20.494831 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:20.494787    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9" podStartSLOduration=2.494771772 podStartE2EDuration="2.494771772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:02:20.492298951 +0000 UTC m=+1617.646377240" watchObservedRunningTime="2026-09-16 15:02:20.494771772 +0000 UTC m=+1617.648850059"
Sep 16 15:02:23.530128 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:23.530102    2558 scope.go:117] "RemoveContainer" containerID="8d2e90798b5aa54ec7629fa0f5fc98707296ca765a99c0f87b407c13aadc0895"
Sep 16 15:02:31.463902 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:31.463870    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:02:31.464413 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:31.463950    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:02:48.147301 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:48.147267    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"]
Sep 16 15:02:48.151157 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:48.151140    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"
Sep 16 15:02:48.154058 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:48.154032    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:02:48.160402 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:48.159878    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"]
Sep 16 15:02:48.229152 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:48.229118    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff87a62-e3be-478b-a9b7-1aab469b4874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-079e6e-smmfk\" (UID: \"aff87a62-e3be-478b-a9b7-1aab469b4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"
Sep 16 15:02:48.330626 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:48.330583    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff87a62-e3be-478b-a9b7-1aab469b4874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-079e6e-smmfk\" (UID: \"aff87a62-e3be-478b-a9b7-1aab469b4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"
Sep 16 15:02:48.331276 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:48.331257    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff87a62-e3be-478b-a9b7-1aab469b4874-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-079e6e-smmfk\" (UID: \"aff87a62-e3be-478b-a9b7-1aab469b4874\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"
Sep 16 15:02:48.464558 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:48.464469    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"
Sep 16 15:02:48.738273 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:48.738247    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"]
Sep 16 15:02:48.740021 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:02:48.739998    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff87a62_e3be_478b_a9b7_1aab469b4874.slice/crio-551aae53ae7d2b24ea6c02b4c1e9b68a5ad80d3c3af6acb903d1d43a7b81ea4e WatchSource:0}: Error finding container 551aae53ae7d2b24ea6c02b4c1e9b68a5ad80d3c3af6acb903d1d43a7b81ea4e: Status 404 returned error can't find the container with id 551aae53ae7d2b24ea6c02b4c1e9b68a5ad80d3c3af6acb903d1d43a7b81ea4e
Sep 16 15:02:49.628772 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:49.628738    2558 generic.go:356] "Generic (PLEG): container finished" podID="aff87a62-e3be-478b-a9b7-1aab469b4874" containerID="c774adccffe428ee718690e10e8fa81322defc9f339b235334b21e5218faa8a2" exitCode=0
Sep 16 15:02:49.629191 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:49.628778    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk" event={"ID":"aff87a62-e3be-478b-a9b7-1aab469b4874","Type":"ContainerDied","Data":"c774adccffe428ee718690e10e8fa81322defc9f339b235334b21e5218faa8a2"}
Sep 16 15:02:49.629191 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:49.628833    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk" event={"ID":"aff87a62-e3be-478b-a9b7-1aab469b4874","Type":"ContainerStarted","Data":"551aae53ae7d2b24ea6c02b4c1e9b68a5ad80d3c3af6acb903d1d43a7b81ea4e"}
Sep 16 15:02:50.769418 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:50.769396    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"
Sep 16 15:02:50.852312 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:50.852284    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff87a62-e3be-478b-a9b7-1aab469b4874-maas-api-service-ca\") pod \"aff87a62-e3be-478b-a9b7-1aab469b4874\" (UID: \"aff87a62-e3be-478b-a9b7-1aab469b4874\") "
Sep 16 15:02:50.852717 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:50.852685    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff87a62-e3be-478b-a9b7-1aab469b4874-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aff87a62-e3be-478b-a9b7-1aab469b4874" (UID: "aff87a62-e3be-478b-a9b7-1aab469b4874"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:02:50.954031 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:50.953948    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff87a62-e3be-478b-a9b7-1aab469b4874-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:02:51.642670 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:51.642591    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"
Sep 16 15:02:51.642670 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:51.642624    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk" event={"ID":"aff87a62-e3be-478b-a9b7-1aab469b4874","Type":"ContainerDied","Data":"551aae53ae7d2b24ea6c02b4c1e9b68a5ad80d3c3af6acb903d1d43a7b81ea4e"}
Sep 16 15:02:51.642670 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:02:51.642669    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="551aae53ae7d2b24ea6c02b4c1e9b68a5ad80d3c3af6acb903d1d43a7b81ea4e"
Sep 16 15:03:03.904687 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:03.904651    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"]
Sep 16 15:03:03.905135 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:03.904903    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx" podUID="3046c826-b1e4-4486-b46b-cc72ef8a51b1" containerName="payload-processing" containerID="cri-o://a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661" gracePeriod=30
Sep 16 15:03:03.915643 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:03.915576    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"]
Sep 16 15:03:03.915938 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:03.915885    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb" podUID="231b64dd-d14a-488f-ae45-2334a423364a" containerName="payload-pre-processing" containerID="cri-o://31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598" gracePeriod=30
Sep 16 15:03:04.137407 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.137352    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"]
Sep 16 15:03:04.141399 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.141332    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-079e6e-smmfk"]
Sep 16 15:03:04.198564 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.198538    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:03:04.201668 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.201648    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:03:04.289673 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.289643    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3046c826-b1e4-4486-b46b-cc72ef8a51b1-plugins-config-volume\") pod \"3046c826-b1e4-4486-b46b-cc72ef8a51b1\" (UID: \"3046c826-b1e4-4486-b46b-cc72ef8a51b1\") "
Sep 16 15:03:04.289865 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.289722    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6q5x\" (UniqueName: \"kubernetes.io/projected/3046c826-b1e4-4486-b46b-cc72ef8a51b1-kube-api-access-s6q5x\") pod \"3046c826-b1e4-4486-b46b-cc72ef8a51b1\" (UID: \"3046c826-b1e4-4486-b46b-cc72ef8a51b1\") "
Sep 16 15:03:04.289865 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.289742    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231b64dd-d14a-488f-ae45-2334a423364a-plugins-config-volume\") pod \"231b64dd-d14a-488f-ae45-2334a423364a\" (UID: \"231b64dd-d14a-488f-ae45-2334a423364a\") "
Sep 16 15:03:04.289865 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.289773    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gvpp\" (UniqueName: \"kubernetes.io/projected/231b64dd-d14a-488f-ae45-2334a423364a-kube-api-access-6gvpp\") pod \"231b64dd-d14a-488f-ae45-2334a423364a\" (UID: \"231b64dd-d14a-488f-ae45-2334a423364a\") "
Sep 16 15:03:04.290091 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.290063    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3046c826-b1e4-4486-b46b-cc72ef8a51b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3046c826-b1e4-4486-b46b-cc72ef8a51b1" (UID: "3046c826-b1e4-4486-b46b-cc72ef8a51b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:03:04.290225 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.290140    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231b64dd-d14a-488f-ae45-2334a423364a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "231b64dd-d14a-488f-ae45-2334a423364a" (UID: "231b64dd-d14a-488f-ae45-2334a423364a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:03:04.291944 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.291918    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231b64dd-d14a-488f-ae45-2334a423364a-kube-api-access-6gvpp" (OuterVolumeSpecName: "kube-api-access-6gvpp") pod "231b64dd-d14a-488f-ae45-2334a423364a" (UID: "231b64dd-d14a-488f-ae45-2334a423364a"). InnerVolumeSpecName "kube-api-access-6gvpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:03:04.292029 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.291956    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3046c826-b1e4-4486-b46b-cc72ef8a51b1-kube-api-access-s6q5x" (OuterVolumeSpecName: "kube-api-access-s6q5x") pod "3046c826-b1e4-4486-b46b-cc72ef8a51b1" (UID: "3046c826-b1e4-4486-b46b-cc72ef8a51b1"). InnerVolumeSpecName "kube-api-access-s6q5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:03:04.391519 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.391481    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gvpp\" (UniqueName: \"kubernetes.io/projected/231b64dd-d14a-488f-ae45-2334a423364a-kube-api-access-6gvpp\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:04.391519 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.391508    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3046c826-b1e4-4486-b46b-cc72ef8a51b1-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:04.391774 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.391537    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6q5x\" (UniqueName: \"kubernetes.io/projected/3046c826-b1e4-4486-b46b-cc72ef8a51b1-kube-api-access-s6q5x\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:04.391774 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.391547    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231b64dd-d14a-488f-ae45-2334a423364a-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:04.716432 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.716399    2558 generic.go:356] "Generic (PLEG): container finished" podID="231b64dd-d14a-488f-ae45-2334a423364a" containerID="31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598" exitCode=0
Sep 16 15:03:04.716622 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.716439    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"
Sep 16 15:03:04.716622 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.716485    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb" event={"ID":"231b64dd-d14a-488f-ae45-2334a423364a","Type":"ContainerDied","Data":"31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598"}
Sep 16 15:03:04.716622 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.716536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb" event={"ID":"231b64dd-d14a-488f-ae45-2334a423364a","Type":"ContainerDied","Data":"b3040e3f86852e67a62eaeaae726290027c4db939ea3df06e2832129bcdace9c"}
Sep 16 15:03:04.716622 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.716563    2558 scope.go:117] "RemoveContainer" containerID="31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598"
Sep 16 15:03:04.718150 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.718128    2558 generic.go:356] "Generic (PLEG): container finished" podID="3046c826-b1e4-4486-b46b-cc72ef8a51b1" containerID="a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661" exitCode=0
Sep 16 15:03:04.718261 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.718184    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx" event={"ID":"3046c826-b1e4-4486-b46b-cc72ef8a51b1","Type":"ContainerDied","Data":"a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661"}
Sep 16 15:03:04.718261 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.718188    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"
Sep 16 15:03:04.718261 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.718228    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx" event={"ID":"3046c826-b1e4-4486-b46b-cc72ef8a51b1","Type":"ContainerDied","Data":"0b6cd9187a87a7b7d51d7715166decce7adf57c8077276cb4f69606712c364fa"}
Sep 16 15:03:04.729238 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.729214    2558 scope.go:117] "RemoveContainer" containerID="31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598"
Sep 16 15:03:04.729742 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:03:04.729616    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598\": container with ID starting with 31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598 not found: ID does not exist" containerID="31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598"
Sep 16 15:03:04.729819 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.729650    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598"} err="failed to get container status \"31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598\": rpc error: code = NotFound desc = could not find container \"31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598\": container with ID starting with 31d1ad36dd8ca0de92eb04701a8d26534f13b7fcaae68a01b40c373f73790598 not found: ID does not exist"
Sep 16 15:03:04.729819 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.729770    2558 scope.go:117] "RemoveContainer" containerID="a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661"
Sep 16 15:03:04.738514 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.738497    2558 scope.go:117] "RemoveContainer" containerID="a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661"
Sep 16 15:03:04.738789 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:03:04.738768    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661\": container with ID starting with a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661 not found: ID does not exist" containerID="a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661"
Sep 16 15:03:04.738860 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.738797    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661"} err="failed to get container status \"a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661\": rpc error: code = NotFound desc = could not find container \"a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661\": container with ID starting with a6a6842808587bd5a0dc48f8f384337de59547439ffaa108b9b1297b22cef661 not found: ID does not exist"
Sep 16 15:03:04.752819 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.752798    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"]
Sep 16 15:03:04.763268 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.763243    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-079e6e-5f7b8758b6-8gfgb"]
Sep 16 15:03:04.780745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.780724    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"]
Sep 16 15:03:04.786923 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:04.786905    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-079e6e-65cffc6767-xsldx"]
Sep 16 15:03:05.296113 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:05.296085    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="231b64dd-d14a-488f-ae45-2334a423364a" path="/var/lib/kubelet/pods/231b64dd-d14a-488f-ae45-2334a423364a/volumes"
Sep 16 15:03:05.296516 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:05.296455    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3046c826-b1e4-4486-b46b-cc72ef8a51b1" path="/var/lib/kubelet/pods/3046c826-b1e4-4486-b46b-cc72ef8a51b1/volumes"
Sep 16 15:03:05.296739 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:05.296726    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aff87a62-e3be-478b-a9b7-1aab469b4874" path="/var/lib/kubelet/pods/aff87a62-e3be-478b-a9b7-1aab469b4874/volumes"
Sep 16 15:03:09.005029 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.004994    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"]
Sep 16 15:03:09.005516 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.005421    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff87a62-e3be-478b-a9b7-1aab469b4874" containerName="revoke-keys"
Sep 16 15:03:09.005516 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.005433    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff87a62-e3be-478b-a9b7-1aab469b4874" containerName="revoke-keys"
Sep 16 15:03:09.005516 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.005442    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="231b64dd-d14a-488f-ae45-2334a423364a" containerName="payload-pre-processing"
Sep 16 15:03:09.005516 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.005447    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="231b64dd-d14a-488f-ae45-2334a423364a" containerName="payload-pre-processing"
Sep 16 15:03:09.005516 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.005463    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3046c826-b1e4-4486-b46b-cc72ef8a51b1" containerName="payload-processing"
Sep 16 15:03:09.005516 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.005469    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3046c826-b1e4-4486-b46b-cc72ef8a51b1" containerName="payload-processing"
Sep 16 15:03:09.005711 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.005525    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="231b64dd-d14a-488f-ae45-2334a423364a" containerName="payload-pre-processing"
Sep 16 15:03:09.005711 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.005535    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3046c826-b1e4-4486-b46b-cc72ef8a51b1" containerName="payload-processing"
Sep 16 15:03:09.005711 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.005543    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff87a62-e3be-478b-a9b7-1aab469b4874" containerName="revoke-keys"
Sep 16 15:03:09.009754 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.009737    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"
Sep 16 15:03:09.012510 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.012491    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:03:09.016515 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.016493    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"]
Sep 16 15:03:09.036920 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.036892    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/152a712b-9e99-4e69-9ecd-8692213ef071-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh\" (UID: \"152a712b-9e99-4e69-9ecd-8692213ef071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"
Sep 16 15:03:09.138026 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.137996    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/152a712b-9e99-4e69-9ecd-8692213ef071-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh\" (UID: \"152a712b-9e99-4e69-9ecd-8692213ef071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"
Sep 16 15:03:09.138650 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.138630    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/152a712b-9e99-4e69-9ecd-8692213ef071-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh\" (UID: \"152a712b-9e99-4e69-9ecd-8692213ef071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"
Sep 16 15:03:09.321739 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.321704    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"
Sep 16 15:03:09.597715 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.597623    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"]
Sep 16 15:03:09.600240 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:03:09.600209    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152a712b_9e99_4e69_9ecd_8692213ef071.slice/crio-3c68b5a0e00d3093178148e41590a1f4af412df83aa68402f6358b8dc196f01d WatchSource:0}: Error finding container 3c68b5a0e00d3093178148e41590a1f4af412df83aa68402f6358b8dc196f01d: Status 404 returned error can't find the container with id 3c68b5a0e00d3093178148e41590a1f4af412df83aa68402f6358b8dc196f01d
Sep 16 15:03:09.749039 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:09.749002    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh" event={"ID":"152a712b-9e99-4e69-9ecd-8692213ef071","Type":"ContainerStarted","Data":"3c68b5a0e00d3093178148e41590a1f4af412df83aa68402f6358b8dc196f01d"}
Sep 16 15:03:10.756409 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:10.756354    2558 generic.go:356] "Generic (PLEG): container finished" podID="152a712b-9e99-4e69-9ecd-8692213ef071" containerID="7df868eb3d6e1cdb34feaf27294037de85d09e40534b724d3e9d82ceab0de253" exitCode=0
Sep 16 15:03:10.756409 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:10.756393    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh" event={"ID":"152a712b-9e99-4e69-9ecd-8692213ef071","Type":"ContainerDied","Data":"7df868eb3d6e1cdb34feaf27294037de85d09e40534b724d3e9d82ceab0de253"}
Sep 16 15:03:11.899934 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:11.899899    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"
Sep 16 15:03:11.965049 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:11.965021    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/152a712b-9e99-4e69-9ecd-8692213ef071-maas-api-service-ca\") pod \"152a712b-9e99-4e69-9ecd-8692213ef071\" (UID: \"152a712b-9e99-4e69-9ecd-8692213ef071\") "
Sep 16 15:03:11.965371 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:11.965348    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152a712b-9e99-4e69-9ecd-8692213ef071-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "152a712b-9e99-4e69-9ecd-8692213ef071" (UID: "152a712b-9e99-4e69-9ecd-8692213ef071"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:03:11.977521 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:11.977490    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"]
Sep 16 15:03:11.977916 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:11.977903    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152a712b-9e99-4e69-9ecd-8692213ef071" containerName="revoke-keys"
Sep 16 15:03:11.977969 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:11.977918    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="152a712b-9e99-4e69-9ecd-8692213ef071" containerName="revoke-keys"
Sep 16 15:03:11.978004 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:11.977992    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="152a712b-9e99-4e69-9ecd-8692213ef071" containerName="revoke-keys"
Sep 16 15:03:11.981265 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:11.981243    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"
Sep 16 15:03:11.991828 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:11.991805    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"]
Sep 16 15:03:12.066679 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:12.066643    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9822349-eb2a-4f2b-ba3d-c6b5521faa3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw\" (UID: \"d9822349-eb2a-4f2b-ba3d-c6b5521faa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"
Sep 16 15:03:12.066865 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:12.066841    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/152a712b-9e99-4e69-9ecd-8692213ef071-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:12.167790 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:12.167748    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9822349-eb2a-4f2b-ba3d-c6b5521faa3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw\" (UID: \"d9822349-eb2a-4f2b-ba3d-c6b5521faa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"
Sep 16 15:03:12.168433 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:12.168412    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9822349-eb2a-4f2b-ba3d-c6b5521faa3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw\" (UID: \"d9822349-eb2a-4f2b-ba3d-c6b5521faa3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"
Sep 16 15:03:12.295776 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:12.295749    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"
Sep 16 15:03:12.576139 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:12.576110    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"]
Sep 16 15:03:12.577756 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:03:12.577726    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9822349_eb2a_4f2b_ba3d_c6b5521faa3b.slice/crio-8ccaf4ad9984c0ab4e5d2bd7249c47c1677b9c36946aa2db37b1fcd86ebae228 WatchSource:0}: Error finding container 8ccaf4ad9984c0ab4e5d2bd7249c47c1677b9c36946aa2db37b1fcd86ebae228: Status 404 returned error can't find the container with id 8ccaf4ad9984c0ab4e5d2bd7249c47c1677b9c36946aa2db37b1fcd86ebae228
Sep 16 15:03:12.768769 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:12.768745    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"
Sep 16 15:03:12.768982 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:12.768746    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh" event={"ID":"152a712b-9e99-4e69-9ecd-8692213ef071","Type":"ContainerDied","Data":"3c68b5a0e00d3093178148e41590a1f4af412df83aa68402f6358b8dc196f01d"}
Sep 16 15:03:12.769096 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:12.769005    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c68b5a0e00d3093178148e41590a1f4af412df83aa68402f6358b8dc196f01d"
Sep 16 15:03:12.770089 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:12.770064    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw" event={"ID":"d9822349-eb2a-4f2b-ba3d-c6b5521faa3b","Type":"ContainerStarted","Data":"8ccaf4ad9984c0ab4e5d2bd7249c47c1677b9c36946aa2db37b1fcd86ebae228"}
Sep 16 15:03:13.777952 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:13.777915    2558 generic.go:356] "Generic (PLEG): container finished" podID="d9822349-eb2a-4f2b-ba3d-c6b5521faa3b" containerID="a061e0034dd839a6d814c0c3b1e776b953d8452b825d05d292f900a40e8a5b19" exitCode=0
Sep 16 15:03:13.778341 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:13.777980    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw" event={"ID":"d9822349-eb2a-4f2b-ba3d-c6b5521faa3b","Type":"ContainerDied","Data":"a061e0034dd839a6d814c0c3b1e776b953d8452b825d05d292f900a40e8a5b19"}
Sep 16 15:03:13.868334 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:13.868300    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"]
Sep 16 15:03:13.887003 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:13.886974    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"]
Sep 16 15:03:13.901485 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:13.901454    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"]
Sep 16 15:03:14.931590 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:14.931564    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"
Sep 16 15:03:15.001476 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:15.001447    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9822349-eb2a-4f2b-ba3d-c6b5521faa3b-maas-api-service-ca\") pod \"d9822349-eb2a-4f2b-ba3d-c6b5521faa3b\" (UID: \"d9822349-eb2a-4f2b-ba3d-c6b5521faa3b\") "
Sep 16 15:03:15.001834 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:15.001810    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9822349-eb2a-4f2b-ba3d-c6b5521faa3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9822349-eb2a-4f2b-ba3d-c6b5521faa3b" (UID: "d9822349-eb2a-4f2b-ba3d-c6b5521faa3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:03:15.102933 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:15.102901    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9822349-eb2a-4f2b-ba3d-c6b5521faa3b-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:15.790263 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:15.790232    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"
Sep 16 15:03:15.790263 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:15.790255    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw" event={"ID":"d9822349-eb2a-4f2b-ba3d-c6b5521faa3b","Type":"ContainerDied","Data":"8ccaf4ad9984c0ab4e5d2bd7249c47c1677b9c36946aa2db37b1fcd86ebae228"}
Sep 16 15:03:15.790470 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:15.790304    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ccaf4ad9984c0ab4e5d2bd7249c47c1677b9c36946aa2db37b1fcd86ebae228"
Sep 16 15:03:23.566573 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:23.566534    2558 scope.go:117] "RemoveContainer" containerID="9b060c70dc1d12cf42a02f7c21c1adbf2e0d2d948da144424c14320680d6d084"
Sep 16 15:03:24.372538 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.372502    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td"]
Sep 16 15:03:24.375141 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.375118    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e33b3f-29826180-252td"]
Sep 16 15:03:24.416058 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.416027    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"]
Sep 16 15:03:24.416348 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.416320    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc" podUID="562dfebc-3b43-488c-affe-0854a5f9faae" containerName="payload-pre-processing" containerID="cri-o://1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a" gracePeriod=30
Sep 16 15:03:24.641737 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.641663    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"]
Sep 16 15:03:24.652511 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.652115    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e33b3f-7xwdh"]
Sep 16 15:03:24.681062 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.681034    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 15:03:24.796409 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.796372    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562dfebc-3b43-488c-affe-0854a5f9faae-plugins-config-volume\") pod \"562dfebc-3b43-488c-affe-0854a5f9faae\" (UID: \"562dfebc-3b43-488c-affe-0854a5f9faae\") "
Sep 16 15:03:24.796570 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.796482    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5r8b\" (UniqueName: \"kubernetes.io/projected/562dfebc-3b43-488c-affe-0854a5f9faae-kube-api-access-x5r8b\") pod \"562dfebc-3b43-488c-affe-0854a5f9faae\" (UID: \"562dfebc-3b43-488c-affe-0854a5f9faae\") "
Sep 16 15:03:24.796740 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.796716    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/562dfebc-3b43-488c-affe-0854a5f9faae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "562dfebc-3b43-488c-affe-0854a5f9faae" (UID: "562dfebc-3b43-488c-affe-0854a5f9faae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:03:24.796870 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.796850    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562dfebc-3b43-488c-affe-0854a5f9faae-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:24.798529 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.798509    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562dfebc-3b43-488c-affe-0854a5f9faae-kube-api-access-x5r8b" (OuterVolumeSpecName: "kube-api-access-x5r8b") pod "562dfebc-3b43-488c-affe-0854a5f9faae" (UID: "562dfebc-3b43-488c-affe-0854a5f9faae"). InnerVolumeSpecName "kube-api-access-x5r8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:03:24.839509 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.839483    2558 generic.go:356] "Generic (PLEG): container finished" podID="562dfebc-3b43-488c-affe-0854a5f9faae" containerID="1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a" exitCode=0
Sep 16 15:03:24.839642 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.839527    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"
Sep 16 15:03:24.839642 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.839574    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc" event={"ID":"562dfebc-3b43-488c-affe-0854a5f9faae","Type":"ContainerDied","Data":"1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a"}
Sep 16 15:03:24.839771 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.839667    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc" event={"ID":"562dfebc-3b43-488c-affe-0854a5f9faae","Type":"ContainerDied","Data":"2c8476333717c79a1b2b79c4ec190981e9dc483b443f120e5751f3f9b5756d58"}
Sep 16 15:03:24.839771 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.839702    2558 scope.go:117] "RemoveContainer" containerID="1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a"
Sep 16 15:03:24.848516 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.848498    2558 scope.go:117] "RemoveContainer" containerID="1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a"
Sep 16 15:03:24.848758 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:03:24.848737    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a\": container with ID starting with 1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a not found: ID does not exist" containerID="1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a"
Sep 16 15:03:24.848807 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.848768    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a"} err="failed to get container status \"1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a\": rpc error: code = NotFound desc = could not find container \"1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a\": container with ID starting with 1725e1c84f141b51dc5bc4d657cde949dbea7dc8e833bde5c0452d08f9ee355a not found: ID does not exist"
Sep 16 15:03:24.861518 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.861492    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"]
Sep 16 15:03:24.868400 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.868380    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e33b3f-7d794d57c4-94tgc"]
Sep 16 15:03:24.897638 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:24.897574    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5r8b\" (UniqueName: \"kubernetes.io/projected/562dfebc-3b43-488c-affe-0854a5f9faae-kube-api-access-x5r8b\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:25.295040 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:25.294961    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="152a712b-9e99-4e69-9ecd-8692213ef071" path="/var/lib/kubelet/pods/152a712b-9e99-4e69-9ecd-8692213ef071/volumes"
Sep 16 15:03:25.295377 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:25.295363    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="562dfebc-3b43-488c-affe-0854a5f9faae" path="/var/lib/kubelet/pods/562dfebc-3b43-488c-affe-0854a5f9faae/volumes"
Sep 16 15:03:25.295608 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:25.295597    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d397300-5b86-441a-a540-f7f7051a4251" path="/var/lib/kubelet/pods/8d397300-5b86-441a-a540-f7f7051a4251/volumes"
Sep 16 15:03:28.887490 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:28.887453    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5" podUID="8995ed84-0e3d-4421-b153-1fe3b487c85b" containerName="main" containerID="cri-o://f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b" gracePeriod=45
Sep 16 15:03:28.907764 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:28.907661    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k" podUID="047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" containerName="main" containerID="cri-o://34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405" gracePeriod=45
Sep 16 15:03:28.920622 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:28.920580    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c" podUID="6c094eca-09ff-4eb2-88ca-17d4996793a9" containerName="main" containerID="cri-o://1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad" gracePeriod=45
Sep 16 15:03:28.921511 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:28.921481    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"]
Sep 16 15:03:28.921825 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:28.921690    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9" podUID="d66bdb3b-72f7-4cda-b56d-5991c516c2d2" containerName="payload-processing" containerID="cri-o://5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b" gracePeriod=30
Sep 16 15:03:28.931946 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:28.931865    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"]
Sep 16 15:03:28.932592 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:28.932082    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq" podUID="26a68ccc-21b8-41e4-80da-1ea60e72fd64" containerName="payload-pre-processing" containerID="cri-o://658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0" gracePeriod=30
Sep 16 15:03:29.108887 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.107857    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"]
Sep 16 15:03:29.111830 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.111766    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-eecd90-qx4vw"]
Sep 16 15:03:29.208450 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.208426    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 15:03:29.258879 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.257571    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:03:29.301050 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.301013    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9822349-eb2a-4f2b-ba3d-c6b5521faa3b" path="/var/lib/kubelet/pods/d9822349-eb2a-4f2b-ba3d-c6b5521faa3b/volumes"
Sep 16 15:03:29.303552 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.303530    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 15:03:29.309101 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.309083    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 15:03:29.312523 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.312503    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:03:29.341201 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.341155    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-dshm\") pod \"8995ed84-0e3d-4421-b153-1fe3b487c85b\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") "
Sep 16 15:03:29.341339 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.341226    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-tmp-dir\") pod \"8995ed84-0e3d-4421-b153-1fe3b487c85b\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") "
Sep 16 15:03:29.341339 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.341252    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5sgn\" (UniqueName: \"kubernetes.io/projected/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-kube-api-access-w5sgn\") pod \"d66bdb3b-72f7-4cda-b56d-5991c516c2d2\" (UID: \"d66bdb3b-72f7-4cda-b56d-5991c516c2d2\") "
Sep 16 15:03:29.341339 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.341301    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8995ed84-0e3d-4421-b153-1fe3b487c85b-tls-certs\") pod \"8995ed84-0e3d-4421-b153-1fe3b487c85b\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") "
Sep 16 15:03:29.341510 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.341376    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-model-cache\") pod \"8995ed84-0e3d-4421-b153-1fe3b487c85b\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") "
Sep 16 15:03:29.341568 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.341496    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8995ed84-0e3d-4421-b153-1fe3b487c85b" (UID: "8995ed84-0e3d-4421-b153-1fe3b487c85b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.341633 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.341566    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8995ed84-0e3d-4421-b153-1fe3b487c85b" (UID: "8995ed84-0e3d-4421-b153-1fe3b487c85b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.341633 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.341628    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d2vw\" (UniqueName: \"kubernetes.io/projected/8995ed84-0e3d-4421-b153-1fe3b487c85b-kube-api-access-2d2vw\") pod \"8995ed84-0e3d-4421-b153-1fe3b487c85b\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") "
Sep 16 15:03:29.341760 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.341654    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-plugins-config-volume\") pod \"d66bdb3b-72f7-4cda-b56d-5991c516c2d2\" (UID: \"d66bdb3b-72f7-4cda-b56d-5991c516c2d2\") "
Sep 16 15:03:29.341760 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.341755    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-home\") pod \"8995ed84-0e3d-4421-b153-1fe3b487c85b\" (UID: \"8995ed84-0e3d-4421-b153-1fe3b487c85b\") "
Sep 16 15:03:29.342129 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.342110    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.342306 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.342292    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.342401 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.342145    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-home" (OuterVolumeSpecName: "home") pod "8995ed84-0e3d-4421-b153-1fe3b487c85b" (UID: "8995ed84-0e3d-4421-b153-1fe3b487c85b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.342486 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.342429    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d66bdb3b-72f7-4cda-b56d-5991c516c2d2" (UID: "d66bdb3b-72f7-4cda-b56d-5991c516c2d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:03:29.343720 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.343689    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-dshm" (OuterVolumeSpecName: "dshm") pod "8995ed84-0e3d-4421-b153-1fe3b487c85b" (UID: "8995ed84-0e3d-4421-b153-1fe3b487c85b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.343809 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.343689    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8995ed84-0e3d-4421-b153-1fe3b487c85b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8995ed84-0e3d-4421-b153-1fe3b487c85b" (UID: "8995ed84-0e3d-4421-b153-1fe3b487c85b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:03:29.343860 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.343847    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8995ed84-0e3d-4421-b153-1fe3b487c85b-kube-api-access-2d2vw" (OuterVolumeSpecName: "kube-api-access-2d2vw") pod "8995ed84-0e3d-4421-b153-1fe3b487c85b" (UID: "8995ed84-0e3d-4421-b153-1fe3b487c85b"). InnerVolumeSpecName "kube-api-access-2d2vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:03:29.344102 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.344082    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-kube-api-access-w5sgn" (OuterVolumeSpecName: "kube-api-access-w5sgn") pod "d66bdb3b-72f7-4cda-b56d-5991c516c2d2" (UID: "d66bdb3b-72f7-4cda-b56d-5991c516c2d2"). InnerVolumeSpecName "kube-api-access-w5sgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:03:29.442942 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.442860    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45jmm\" (UniqueName: \"kubernetes.io/projected/6c094eca-09ff-4eb2-88ca-17d4996793a9-kube-api-access-45jmm\") pod \"6c094eca-09ff-4eb2-88ca-17d4996793a9\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") "
Sep 16 15:03:29.442942 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.442900    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vxkw\" (UniqueName: \"kubernetes.io/projected/26a68ccc-21b8-41e4-80da-1ea60e72fd64-kube-api-access-9vxkw\") pod \"26a68ccc-21b8-41e4-80da-1ea60e72fd64\" (UID: \"26a68ccc-21b8-41e4-80da-1ea60e72fd64\") "
Sep 16 15:03:29.442942 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.442930    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-model-cache\") pod \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") "
Sep 16 15:03:29.443145 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443096    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-home\") pod \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") "
Sep 16 15:03:29.443145 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443137    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c094eca-09ff-4eb2-88ca-17d4996793a9-tls-certs\") pod \"6c094eca-09ff-4eb2-88ca-17d4996793a9\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") "
Sep 16 15:03:29.443259 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443154    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-model-cache" (OuterVolumeSpecName: "model-cache") pod "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" (UID: "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.443259 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443208    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-dshm\") pod \"6c094eca-09ff-4eb2-88ca-17d4996793a9\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") "
Sep 16 15:03:29.443259 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443229    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tls-certs\") pod \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") "
Sep 16 15:03:29.443414 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443265    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-tmp-dir\") pod \"6c094eca-09ff-4eb2-88ca-17d4996793a9\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") "
Sep 16 15:03:29.443414 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443311    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tmp-dir\") pod \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") "
Sep 16 15:03:29.443414 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443356    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-model-cache\") pod \"6c094eca-09ff-4eb2-88ca-17d4996793a9\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") "
Sep 16 15:03:29.443414 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443372    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-home" (OuterVolumeSpecName: "home") pod "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" (UID: "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.443414 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443382    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7svf4\" (UniqueName: \"kubernetes.io/projected/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-kube-api-access-7svf4\") pod \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") "
Sep 16 15:03:29.443682 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443432    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-home\") pod \"6c094eca-09ff-4eb2-88ca-17d4996793a9\" (UID: \"6c094eca-09ff-4eb2-88ca-17d4996793a9\") "
Sep 16 15:03:29.443682 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443522    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-dshm\") pod \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\" (UID: \"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18\") "
Sep 16 15:03:29.443682 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443539    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c094eca-09ff-4eb2-88ca-17d4996793a9" (UID: "6c094eca-09ff-4eb2-88ca-17d4996793a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.443682 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443552    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a68ccc-21b8-41e4-80da-1ea60e72fd64-plugins-config-volume\") pod \"26a68ccc-21b8-41e4-80da-1ea60e72fd64\" (UID: \"26a68ccc-21b8-41e4-80da-1ea60e72fd64\") "
Sep 16 15:03:29.443891 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443744    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-home" (OuterVolumeSpecName: "home") pod "6c094eca-09ff-4eb2-88ca-17d4996793a9" (UID: "6c094eca-09ff-4eb2-88ca-17d4996793a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.444007 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.443988    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.444079 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444011    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.444079 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444025    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8995ed84-0e3d-4421-b153-1fe3b487c85b-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.444079 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444038    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5sgn\" (UniqueName: \"kubernetes.io/projected/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-kube-api-access-w5sgn\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.444079 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444051    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.444079 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444064    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8995ed84-0e3d-4421-b153-1fe3b487c85b-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.444079 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444077    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d2vw\" (UniqueName: \"kubernetes.io/projected/8995ed84-0e3d-4421-b153-1fe3b487c85b-kube-api-access-2d2vw\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.444422 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444077    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a68ccc-21b8-41e4-80da-1ea60e72fd64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "26a68ccc-21b8-41e4-80da-1ea60e72fd64" (UID: "26a68ccc-21b8-41e4-80da-1ea60e72fd64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:03:29.444422 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444092    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d66bdb3b-72f7-4cda-b56d-5991c516c2d2-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.444422 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444107    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.444422 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444120    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.444422 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444341    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" (UID: "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.444679 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.444521    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c094eca-09ff-4eb2-88ca-17d4996793a9" (UID: "6c094eca-09ff-4eb2-88ca-17d4996793a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.445573 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.445542    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" (UID: "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:03:29.445573 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.445546    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a68ccc-21b8-41e4-80da-1ea60e72fd64-kube-api-access-9vxkw" (OuterVolumeSpecName: "kube-api-access-9vxkw") pod "26a68ccc-21b8-41e4-80da-1ea60e72fd64" (UID: "26a68ccc-21b8-41e4-80da-1ea60e72fd64"). InnerVolumeSpecName "kube-api-access-9vxkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:03:29.445804 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.445786    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c094eca-09ff-4eb2-88ca-17d4996793a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c094eca-09ff-4eb2-88ca-17d4996793a9" (UID: "6c094eca-09ff-4eb2-88ca-17d4996793a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:03:29.445866 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.445848    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-kube-api-access-7svf4" (OuterVolumeSpecName: "kube-api-access-7svf4") pod "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" (UID: "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18"). InnerVolumeSpecName "kube-api-access-7svf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:03:29.445918 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.445902    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-dshm" (OuterVolumeSpecName: "dshm") pod "6c094eca-09ff-4eb2-88ca-17d4996793a9" (UID: "6c094eca-09ff-4eb2-88ca-17d4996793a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.446277 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.446258    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-dshm" (OuterVolumeSpecName: "dshm") pod "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" (UID: "047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:03:29.446344 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.446281    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c094eca-09ff-4eb2-88ca-17d4996793a9-kube-api-access-45jmm" (OuterVolumeSpecName: "kube-api-access-45jmm") pod "6c094eca-09ff-4eb2-88ca-17d4996793a9" (UID: "6c094eca-09ff-4eb2-88ca-17d4996793a9"). InnerVolumeSpecName "kube-api-access-45jmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:03:29.545526 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.545500    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.545526 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.545524    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26a68ccc-21b8-41e4-80da-1ea60e72fd64-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.545526 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.545534    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45jmm\" (UniqueName: \"kubernetes.io/projected/6c094eca-09ff-4eb2-88ca-17d4996793a9-kube-api-access-45jmm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.545762 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.545544    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vxkw\" (UniqueName: \"kubernetes.io/projected/26a68ccc-21b8-41e4-80da-1ea60e72fd64-kube-api-access-9vxkw\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.545762 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.545552    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c094eca-09ff-4eb2-88ca-17d4996793a9-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.545762 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.545560    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.545762 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.545567    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.545762 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.545574    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.545762 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.545581    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c094eca-09ff-4eb2-88ca-17d4996793a9-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.545762 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.545589    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7svf4\" (UniqueName: \"kubernetes.io/projected/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18-kube-api-access-7svf4\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:29.869052 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.869021    2558 generic.go:356] "Generic (PLEG): container finished" podID="047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" containerID="34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405" exitCode=0
Sep 16 15:03:29.869261 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.869074    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"
Sep 16 15:03:29.869261 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.869102    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k" event={"ID":"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18","Type":"ContainerDied","Data":"34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405"}
Sep 16 15:03:29.869261 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.869142    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k" event={"ID":"047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18","Type":"ContainerDied","Data":"45304fa588c5bb5e5e4f4cba1097835c74f61e6f7c72cf9a285320484e3c8baf"}
Sep 16 15:03:29.869261 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.869193    2558 scope.go:117] "RemoveContainer" containerID="34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405"
Sep 16 15:03:29.870423 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.870395    2558 generic.go:356] "Generic (PLEG): container finished" podID="26a68ccc-21b8-41e4-80da-1ea60e72fd64" containerID="658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0" exitCode=0
Sep 16 15:03:29.870543 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.870425    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq" event={"ID":"26a68ccc-21b8-41e4-80da-1ea60e72fd64","Type":"ContainerDied","Data":"658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0"}
Sep 16 15:03:29.870543 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.870462    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"
Sep 16 15:03:29.870543 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.870499    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq" event={"ID":"26a68ccc-21b8-41e4-80da-1ea60e72fd64","Type":"ContainerDied","Data":"a952006af5676cbd6724c9c9d3e6c0206fb11cf47934130866840f6124544e1e"}
Sep 16 15:03:29.871959 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.871938    2558 generic.go:356] "Generic (PLEG): container finished" podID="d66bdb3b-72f7-4cda-b56d-5991c516c2d2" containerID="5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b" exitCode=0
Sep 16 15:03:29.872070 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.871964    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9" event={"ID":"d66bdb3b-72f7-4cda-b56d-5991c516c2d2","Type":"ContainerDied","Data":"5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b"}
Sep 16 15:03:29.872070 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.872007    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9" event={"ID":"d66bdb3b-72f7-4cda-b56d-5991c516c2d2","Type":"ContainerDied","Data":"46af1eb4ae03b9b82cdc20090747b3c8fd5d95821bb01876faebd48fe9fa20b4"}
Sep 16 15:03:29.872070 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.872019    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"
Sep 16 15:03:29.874719 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.874685    2558 generic.go:356] "Generic (PLEG): container finished" podID="6c094eca-09ff-4eb2-88ca-17d4996793a9" containerID="1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad" exitCode=0
Sep 16 15:03:29.874827 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.874736    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"
Sep 16 15:03:29.874827 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.874765    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c" event={"ID":"6c094eca-09ff-4eb2-88ca-17d4996793a9","Type":"ContainerDied","Data":"1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad"}
Sep 16 15:03:29.874827 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.874796    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c" event={"ID":"6c094eca-09ff-4eb2-88ca-17d4996793a9","Type":"ContainerDied","Data":"cc17d9e67a8417ad0be6551a767d5688474be64d3ca798f02a1ba88464bd6cfc"}
Sep 16 15:03:29.876329 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.876304    2558 generic.go:356] "Generic (PLEG): container finished" podID="8995ed84-0e3d-4421-b153-1fe3b487c85b" containerID="f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b" exitCode=0
Sep 16 15:03:29.876439 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.876373    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5" event={"ID":"8995ed84-0e3d-4421-b153-1fe3b487c85b","Type":"ContainerDied","Data":"f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b"}
Sep 16 15:03:29.876439 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.876400    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5" event={"ID":"8995ed84-0e3d-4421-b153-1fe3b487c85b","Type":"ContainerDied","Data":"1f25f6ec9e654997824c3fc7549151b6e37b814234c211b4b1d112fcfac7de04"}
Sep 16 15:03:29.876553 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.876467    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"
Sep 16 15:03:29.880601 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.880568    2558 scope.go:117] "RemoveContainer" containerID="34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405"
Sep 16 15:03:29.880916 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:03:29.880891    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405\": container with ID starting with 34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405 not found: ID does not exist" containerID="34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405"
Sep 16 15:03:29.880994 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.880928    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405"} err="failed to get container status \"34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405\": rpc error: code = NotFound desc = could not find container \"34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405\": container with ID starting with 34473ee82240a623e88069e727270e867a95e5f02ce2d5856bd7ab8ea4497405 not found: ID does not exist"
Sep 16 15:03:29.880994 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.880948    2558 scope.go:117] "RemoveContainer" containerID="658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0"
Sep 16 15:03:29.891250 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.891007    2558 scope.go:117] "RemoveContainer" containerID="658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0"
Sep 16 15:03:29.891475 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:03:29.891349    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0\": container with ID starting with 658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0 not found: ID does not exist" containerID="658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0"
Sep 16 15:03:29.891475 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.891370    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0"} err="failed to get container status \"658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0\": rpc error: code = NotFound desc = could not find container \"658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0\": container with ID starting with 658c7f67e4d859c00679d12577c54bd8de7507d443134c23f87c0b92dadd40c0 not found: ID does not exist"
Sep 16 15:03:29.891475 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.891385    2558 scope.go:117] "RemoveContainer" containerID="5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b"
Sep 16 15:03:29.901302 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.901249    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"]
Sep 16 15:03:29.908107 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.908088    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-eecd90-79b6b476fd-x8njq"]
Sep 16 15:03:29.919624 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.919604    2558 scope.go:117] "RemoveContainer" containerID="5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b"
Sep 16 15:03:29.919921 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:03:29.919899    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b\": container with ID starting with 5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b not found: ID does not exist" containerID="5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b"
Sep 16 15:03:29.920000 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.919932    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b"} err="failed to get container status \"5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b\": rpc error: code = NotFound desc = could not find container \"5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b\": container with ID starting with 5ecbcf93aadd3db5a23ca85e53beb3a4a64af74aafb177050c9ef4a531ee253b not found: ID does not exist"
Sep 16 15:03:29.920000 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.919972    2558 scope.go:117] "RemoveContainer" containerID="1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad"
Sep 16 15:03:29.921375 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.921356    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"]
Sep 16 15:03:29.925663 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.925643    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/premium-w4-e33b3f-kserve-688d464c5b-d7t4k"]
Sep 16 15:03:29.928390 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.928374    2558 scope.go:117] "RemoveContainer" containerID="1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad"
Sep 16 15:03:29.928657 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:03:29.928635    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad\": container with ID starting with 1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad not found: ID does not exist" containerID="1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad"
Sep 16 15:03:29.928736 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.928661    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad"} err="failed to get container status \"1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad\": rpc error: code = NotFound desc = could not find container \"1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad\": container with ID starting with 1fe5514828c72661e70dce589eabd46963feb6a6672227d43da5d1ad2ac809ad not found: ID does not exist"
Sep 16 15:03:29.928736 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.928677    2558 scope.go:117] "RemoveContainer" containerID="f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b"
Sep 16 15:03:29.936306 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.936286    2558 scope.go:117] "RemoveContainer" containerID="f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b"
Sep 16 15:03:29.936577 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:03:29.936560    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b\": container with ID starting with f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b not found: ID does not exist" containerID="f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b"
Sep 16 15:03:29.936645 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.936581    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b"} err="failed to get container status \"f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b\": rpc error: code = NotFound desc = could not find container \"f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b\": container with ID starting with f58b5b02b457ca0f5c78da1a1322aea534e4daae7f8059179acbd40629ab139b not found: ID does not exist"
Sep 16 15:03:29.937048 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.937031    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"]
Sep 16 15:03:29.942651 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.942632    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/distinct-2-w4-e33b3f-kserve-5689846856-dxgh5"]
Sep 16 15:03:29.957419 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.957399    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"]
Sep 16 15:03:29.964885 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.964864    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e33b3f/simulator-w4-e33b3f-kserve-786b56b8fc-qhj5c"]
Sep 16 15:03:29.976687 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.976665    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"]
Sep 16 15:03:29.982013 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:29.981994    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-eecd90-5f56c75db5-ccfs9"]
Sep 16 15:03:31.295463 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:31.295430    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" path="/var/lib/kubelet/pods/047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18/volumes"
Sep 16 15:03:31.295947 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:31.295931    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26a68ccc-21b8-41e4-80da-1ea60e72fd64" path="/var/lib/kubelet/pods/26a68ccc-21b8-41e4-80da-1ea60e72fd64/volumes"
Sep 16 15:03:31.296443 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:31.296427    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c094eca-09ff-4eb2-88ca-17d4996793a9" path="/var/lib/kubelet/pods/6c094eca-09ff-4eb2-88ca-17d4996793a9/volumes"
Sep 16 15:03:31.296941 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:31.296921    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8995ed84-0e3d-4421-b153-1fe3b487c85b" path="/var/lib/kubelet/pods/8995ed84-0e3d-4421-b153-1fe3b487c85b/volumes"
Sep 16 15:03:31.297333 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:31.297317    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66bdb3b-72f7-4cda-b56d-5991c516c2d2" path="/var/lib/kubelet/pods/d66bdb3b-72f7-4cda-b56d-5991c516c2d2/volumes"
Sep 16 15:03:36.569247 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569210    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"]
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569578    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66bdb3b-72f7-4cda-b56d-5991c516c2d2" containerName="payload-processing"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569588    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66bdb3b-72f7-4cda-b56d-5991c516c2d2" containerName="payload-processing"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569603    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c094eca-09ff-4eb2-88ca-17d4996793a9" containerName="main"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569608    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c094eca-09ff-4eb2-88ca-17d4996793a9" containerName="main"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569615    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" containerName="main"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569622    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" containerName="main"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569629    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8995ed84-0e3d-4421-b153-1fe3b487c85b" containerName="main"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569634    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8995ed84-0e3d-4421-b153-1fe3b487c85b" containerName="main"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569651    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="562dfebc-3b43-488c-affe-0854a5f9faae" containerName="payload-pre-processing"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569656    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="562dfebc-3b43-488c-affe-0854a5f9faae" containerName="payload-pre-processing"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569661    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9822349-eb2a-4f2b-ba3d-c6b5521faa3b" containerName="revoke-keys"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569666    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9822349-eb2a-4f2b-ba3d-c6b5521faa3b" containerName="revoke-keys"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569672    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a68ccc-21b8-41e4-80da-1ea60e72fd64" containerName="payload-pre-processing"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569677    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a68ccc-21b8-41e4-80da-1ea60e72fd64" containerName="payload-pre-processing"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569727    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c094eca-09ff-4eb2-88ca-17d4996793a9" containerName="main"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569734    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9822349-eb2a-4f2b-ba3d-c6b5521faa3b" containerName="revoke-keys"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569740    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="562dfebc-3b43-488c-affe-0854a5f9faae" containerName="payload-pre-processing"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569746    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66bdb3b-72f7-4cda-b56d-5991c516c2d2" containerName="payload-processing"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569754    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="047ddc24-2ba8-4e95-b6f6-a3fa6c4a4b18" containerName="main"
Sep 16 15:03:36.569745 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569760    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8995ed84-0e3d-4421-b153-1fe3b487c85b" containerName="main"
Sep 16 15:03:36.570792 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.569765    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a68ccc-21b8-41e4-80da-1ea60e72fd64" containerName="payload-pre-processing"
Sep 16 15:03:36.574601 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.574576    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"
Sep 16 15:03:36.577701 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.577683    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:03:36.588672 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.588652    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"]
Sep 16 15:03:36.614673 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.614649    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd0643a-96a6-4f14-8a0e-0b1f37729da6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g\" (UID: \"ccd0643a-96a6-4f14-8a0e-0b1f37729da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"
Sep 16 15:03:36.715561 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.715528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd0643a-96a6-4f14-8a0e-0b1f37729da6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g\" (UID: \"ccd0643a-96a6-4f14-8a0e-0b1f37729da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"
Sep 16 15:03:36.716124 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.716106    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd0643a-96a6-4f14-8a0e-0b1f37729da6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g\" (UID: \"ccd0643a-96a6-4f14-8a0e-0b1f37729da6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"
Sep 16 15:03:36.885144 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:36.885062    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"
Sep 16 15:03:37.163428 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:37.163405    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"]
Sep 16 15:03:37.165986 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:03:37.165956    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd0643a_96a6_4f14_8a0e_0b1f37729da6.slice/crio-31e62e5e364adc5ab8e4e95e21fd423020609cdc84d47b70a39636dbc760baf2 WatchSource:0}: Error finding container 31e62e5e364adc5ab8e4e95e21fd423020609cdc84d47b70a39636dbc760baf2: Status 404 returned error can't find the container with id 31e62e5e364adc5ab8e4e95e21fd423020609cdc84d47b70a39636dbc760baf2
Sep 16 15:03:37.930346 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:37.930314    2558 generic.go:356] "Generic (PLEG): container finished" podID="ccd0643a-96a6-4f14-8a0e-0b1f37729da6" containerID="5054e31ae469e82c17a685aa3904580be17326f892146be85516a903c688caf5" exitCode=0
Sep 16 15:03:37.930759 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:37.930408    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g" event={"ID":"ccd0643a-96a6-4f14-8a0e-0b1f37729da6","Type":"ContainerDied","Data":"5054e31ae469e82c17a685aa3904580be17326f892146be85516a903c688caf5"}
Sep 16 15:03:37.930759 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:37.930477    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g" event={"ID":"ccd0643a-96a6-4f14-8a0e-0b1f37729da6","Type":"ContainerStarted","Data":"31e62e5e364adc5ab8e4e95e21fd423020609cdc84d47b70a39636dbc760baf2"}
Sep 16 15:03:39.058225 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:39.058200    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"
Sep 16 15:03:39.140118 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:39.140092    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd0643a-96a6-4f14-8a0e-0b1f37729da6-maas-api-service-ca\") pod \"ccd0643a-96a6-4f14-8a0e-0b1f37729da6\" (UID: \"ccd0643a-96a6-4f14-8a0e-0b1f37729da6\") "
Sep 16 15:03:39.140489 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:39.140468    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd0643a-96a6-4f14-8a0e-0b1f37729da6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ccd0643a-96a6-4f14-8a0e-0b1f37729da6" (UID: "ccd0643a-96a6-4f14-8a0e-0b1f37729da6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:03:39.240916 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:39.240847    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccd0643a-96a6-4f14-8a0e-0b1f37729da6-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:39.942510 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:39.942470    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g" event={"ID":"ccd0643a-96a6-4f14-8a0e-0b1f37729da6","Type":"ContainerDied","Data":"31e62e5e364adc5ab8e4e95e21fd423020609cdc84d47b70a39636dbc760baf2"}
Sep 16 15:03:39.942699 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:39.942531    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31e62e5e364adc5ab8e4e95e21fd423020609cdc84d47b70a39636dbc760baf2"
Sep 16 15:03:39.942699 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:39.942494    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"
Sep 16 15:03:45.911480 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:45.911405    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7676-3815b9"]
Sep 16 15:03:45.911837 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:45.911790    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccd0643a-96a6-4f14-8a0e-0b1f37729da6" containerName="revoke-keys"
Sep 16 15:03:45.911837 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:45.911801    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccd0643a-96a6-4f14-8a0e-0b1f37729da6" containerName="revoke-keys"
Sep 16 15:03:45.911914 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:45.911866    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccd0643a-96a6-4f14-8a0e-0b1f37729da6" containerName="revoke-keys"
Sep 16 15:03:45.915138 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:45.915121    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-3815b9"
Sep 16 15:03:45.918612 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:45.918585    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x6cz7\""
Sep 16 15:03:45.920349 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:45.920328    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7676-3815b9"]
Sep 16 15:03:46.002913 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:46.002881    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68r6\" (UniqueName: \"kubernetes.io/projected/5c4e0f3a-af55-4487-ba99-bd3e9cc547a8-kube-api-access-h68r6\") pod \"test-curl-7676-3815b9\" (UID: \"5c4e0f3a-af55-4487-ba99-bd3e9cc547a8\") " pod="opendatahub/test-curl-7676-3815b9"
Sep 16 15:03:46.103743 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:46.103715    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68r6\" (UniqueName: \"kubernetes.io/projected/5c4e0f3a-af55-4487-ba99-bd3e9cc547a8-kube-api-access-h68r6\") pod \"test-curl-7676-3815b9\" (UID: \"5c4e0f3a-af55-4487-ba99-bd3e9cc547a8\") " pod="opendatahub/test-curl-7676-3815b9"
Sep 16 15:03:46.114442 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:46.114413    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68r6\" (UniqueName: \"kubernetes.io/projected/5c4e0f3a-af55-4487-ba99-bd3e9cc547a8-kube-api-access-h68r6\") pod \"test-curl-7676-3815b9\" (UID: \"5c4e0f3a-af55-4487-ba99-bd3e9cc547a8\") " pod="opendatahub/test-curl-7676-3815b9"
Sep 16 15:03:46.225311 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:46.225239    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-3815b9"
Sep 16 15:03:46.498994 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:46.498969    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7676-3815b9"]
Sep 16 15:03:46.501187 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:03:46.501140    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c4e0f3a_af55_4487_ba99_bd3e9cc547a8.slice/crio-45548185ef5edda4cb0ffee25349cd5675d11d6b8f922a6eb15378ffa4c35336 WatchSource:0}: Error finding container 45548185ef5edda4cb0ffee25349cd5675d11d6b8f922a6eb15378ffa4c35336: Status 404 returned error can't find the container with id 45548185ef5edda4cb0ffee25349cd5675d11d6b8f922a6eb15378ffa4c35336
Sep 16 15:03:46.503351 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:46.503335    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 15:03:46.981905 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:46.981870    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7676-3815b9" event={"ID":"5c4e0f3a-af55-4487-ba99-bd3e9cc547a8","Type":"ContainerStarted","Data":"45548185ef5edda4cb0ffee25349cd5675d11d6b8f922a6eb15378ffa4c35336"}
Sep 16 15:03:47.405564 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:47.405531    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j"]
Sep 16 15:03:47.413938 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:47.413906    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-05b216-29826180-jmz2j"]
Sep 16 15:03:47.859812 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:47.859775    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"]
Sep 16 15:03:47.862415 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:47.862388    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-05b216-zqs6g"]
Sep 16 15:03:48.997602 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:48.997512    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7676-3815b9" event={"ID":"5c4e0f3a-af55-4487-ba99-bd3e9cc547a8","Type":"ContainerStarted","Data":"4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00"}
Sep 16 15:03:49.019517 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:49.019474    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7676-3815b9" podStartSLOduration=1.900458493 podStartE2EDuration="4.019458816s" totalImagesPullingTime="2.119000323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:03:49.016437591 +0000 UTC m=+1706.170515879" watchObservedRunningTime="2026-09-16 15:03:49.019458816 +0000 UTC m=+1706.173537102"
Sep 16 15:03:49.296418 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:49.296333    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a940beb6-1ae2-4774-ac89-33605adaa06d" path="/var/lib/kubelet/pods/a940beb6-1ae2-4774-ac89-33605adaa06d/volumes"
Sep 16 15:03:49.296749 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:49.296731    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccd0643a-96a6-4f14-8a0e-0b1f37729da6" path="/var/lib/kubelet/pods/ccd0643a-96a6-4f14-8a0e-0b1f37729da6/volumes"
Sep 16 15:03:49.671084 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:49.671044    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7676-3815b9"]
Sep 16 15:03:49.685030 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:49.685008    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7676-3815b9"]
Sep 16 15:03:50.003098 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:03:50.003016    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7676-3815b9" container="test-curl-7676-3815b9"
Sep 16 15:03:51.008584 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:51.008542    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7676-3815b9" podUID="5c4e0f3a-af55-4487-ba99-bd3e9cc547a8" containerName="test-curl-7676-3815b9" containerID="cri-o://4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00" gracePeriod=2
Sep 16 15:03:51.011613 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:03:51.011586    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7676-3815b9\" is forbidden: User \"system:node:ip-10-0-128-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-123.ec2.internal' and this object" podUID="5c4e0f3a-af55-4487-ba99-bd3e9cc547a8" pod="opendatahub/test-curl-7676-3815b9"
Sep 16 15:03:52.776684 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:52.776651    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"]
Sep 16 15:03:53.146688 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:53.146664    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-3815b9"
Sep 16 15:03:53.268886 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:53.268852    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h68r6\" (UniqueName: \"kubernetes.io/projected/5c4e0f3a-af55-4487-ba99-bd3e9cc547a8-kube-api-access-h68r6\") pod \"5c4e0f3a-af55-4487-ba99-bd3e9cc547a8\" (UID: \"5c4e0f3a-af55-4487-ba99-bd3e9cc547a8\") "
Sep 16 15:03:53.270876 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:53.270840    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4e0f3a-af55-4487-ba99-bd3e9cc547a8-kube-api-access-h68r6" (OuterVolumeSpecName: "kube-api-access-h68r6") pod "5c4e0f3a-af55-4487-ba99-bd3e9cc547a8" (UID: "5c4e0f3a-af55-4487-ba99-bd3e9cc547a8"). InnerVolumeSpecName "kube-api-access-h68r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:03:53.295059 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:53.295002    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c4e0f3a-af55-4487-ba99-bd3e9cc547a8" path="/var/lib/kubelet/pods/5c4e0f3a-af55-4487-ba99-bd3e9cc547a8/volumes"
Sep 16 15:03:53.369684 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:53.369660    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h68r6\" (UniqueName: \"kubernetes.io/projected/5c4e0f3a-af55-4487-ba99-bd3e9cc547a8-kube-api-access-h68r6\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:54.026635 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:54.026600    2558 generic.go:356] "Generic (PLEG): container finished" podID="5c4e0f3a-af55-4487-ba99-bd3e9cc547a8" containerID="4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00" exitCode=137
Sep 16 15:03:54.026635 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:54.026638    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-3815b9"
Sep 16 15:03:54.027203 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:54.026695    2558 scope.go:117] "RemoveContainer" containerID="4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00"
Sep 16 15:03:54.030085 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:54.030061    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"]
Sep 16 15:03:54.036390 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:54.036371    2558 scope.go:117] "RemoveContainer" containerID="4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00"
Sep 16 15:03:54.036645 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:03:54.036629    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00\": container with ID starting with 4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00 not found: ID does not exist" containerID="4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00"
Sep 16 15:03:54.036691 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:54.036651    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00"} err="failed to get container status \"4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00\": rpc error: code = NotFound desc = could not find container \"4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00\": container with ID starting with 4339ce56f2a96ff0887d91aab28a1ba17f7e9ac79b039d77a26bfcab5efeee00 not found: ID does not exist"
Sep 16 15:03:55.488563 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:55.488531    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"]
Sep 16 15:03:55.489036 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:55.488892    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4e0f3a-af55-4487-ba99-bd3e9cc547a8" containerName="test-curl-7676-3815b9"
Sep 16 15:03:55.489036 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:55.488903    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4e0f3a-af55-4487-ba99-bd3e9cc547a8" containerName="test-curl-7676-3815b9"
Sep 16 15:03:55.489036 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:55.488968    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c4e0f3a-af55-4487-ba99-bd3e9cc547a8" containerName="test-curl-7676-3815b9"
Sep 16 15:03:55.491899 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:55.491883    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"
Sep 16 15:03:55.495543 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:55.495517    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:03:55.591330 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:55.591299    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5e46881-e473-442a-aa09-9f40ba9c017f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5\" (UID: \"f5e46881-e473-442a-aa09-9f40ba9c017f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"
Sep 16 15:03:55.598855 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:55.598815    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"]
Sep 16 15:03:55.692568 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:55.692533    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5e46881-e473-442a-aa09-9f40ba9c017f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5\" (UID: \"f5e46881-e473-442a-aa09-9f40ba9c017f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"
Sep 16 15:03:55.693127 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:55.693108    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5e46881-e473-442a-aa09-9f40ba9c017f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5\" (UID: \"f5e46881-e473-442a-aa09-9f40ba9c017f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"
Sep 16 15:03:55.801297 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:55.801217    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"
Sep 16 15:03:56.096699 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:56.096670    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"]
Sep 16 15:03:56.097811 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:03:56.097783    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e46881_e473_442a_aa09_9f40ba9c017f.slice/crio-aba9f815f8d634bfd5640ff7b4119821ff002072b5322cdab9dfddc2be3e3dd6 WatchSource:0}: Error finding container aba9f815f8d634bfd5640ff7b4119821ff002072b5322cdab9dfddc2be3e3dd6: Status 404 returned error can't find the container with id aba9f815f8d634bfd5640ff7b4119821ff002072b5322cdab9dfddc2be3e3dd6
Sep 16 15:03:57.045570 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:57.045483    2558 generic.go:356] "Generic (PLEG): container finished" podID="f5e46881-e473-442a-aa09-9f40ba9c017f" containerID="df2d4e8ec34a4c68c92a72e2be3beb8d2dbeea35f21804764cfb02e1c19e4054" exitCode=0
Sep 16 15:03:57.045933 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:57.045563    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5" event={"ID":"f5e46881-e473-442a-aa09-9f40ba9c017f","Type":"ContainerDied","Data":"df2d4e8ec34a4c68c92a72e2be3beb8d2dbeea35f21804764cfb02e1c19e4054"}
Sep 16 15:03:57.045933 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:57.045627    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5" event={"ID":"f5e46881-e473-442a-aa09-9f40ba9c017f","Type":"ContainerStarted","Data":"aba9f815f8d634bfd5640ff7b4119821ff002072b5322cdab9dfddc2be3e3dd6"}
Sep 16 15:03:57.927186 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:57.927127    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"]
Sep 16 15:03:57.930480 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:57.930462    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:57.933324 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:57.933300    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d28bd5\"/\"sub-test-model-d28bd5-kserve-self-signed-certs\""
Sep 16 15:03:57.943268 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:57.943247    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"]
Sep 16 15:03:58.016343 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.016312    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-dshm\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.016506 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.016369    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e084c0-234d-42b4-a30a-891581247013-tls-certs\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.016506 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.016400    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-model-cache\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.016506 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.016418    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccb8p\" (UniqueName: \"kubernetes.io/projected/80e084c0-234d-42b4-a30a-891581247013-kube-api-access-ccb8p\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.016506 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.016474    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-home\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.016640 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.016534    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-tmp-dir\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.117815 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.117776    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-model-cache\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.118247 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.117821    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccb8p\" (UniqueName: \"kubernetes.io/projected/80e084c0-234d-42b4-a30a-891581247013-kube-api-access-ccb8p\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.118247 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.117912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-home\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.118247 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.117953    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-tmp-dir\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.118247 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.117994    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-dshm\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.118247 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.118042    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e084c0-234d-42b4-a30a-891581247013-tls-certs\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.118538 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.118251    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-model-cache\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.118538 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.118336    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-tmp-dir\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.118672 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.118651    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-home\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.120391 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.120358    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-dshm\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.120988 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.120969    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e084c0-234d-42b4-a30a-891581247013-tls-certs\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.126845 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.126810    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccb8p\" (UniqueName: \"kubernetes.io/projected/80e084c0-234d-42b4-a30a-891581247013-kube-api-access-ccb8p\") pod \"sub-test-model-d28bd5-kserve-74b647b949-gljsh\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.185411 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.185359    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"
Sep 16 15:03:58.239955 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.239925    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:03:58.319612 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.319585    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5e46881-e473-442a-aa09-9f40ba9c017f-maas-api-service-ca\") pod \"f5e46881-e473-442a-aa09-9f40ba9c017f\" (UID: \"f5e46881-e473-442a-aa09-9f40ba9c017f\") "
Sep 16 15:03:58.319923 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.319901    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e46881-e473-442a-aa09-9f40ba9c017f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5e46881-e473-442a-aa09-9f40ba9c017f" (UID: "f5e46881-e473-442a-aa09-9f40ba9c017f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:03:58.421211 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.421137    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5e46881-e473-442a-aa09-9f40ba9c017f-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:03:58.527848 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:58.527729    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"]
Sep 16 15:03:58.530578 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:03:58.530556    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80e084c0_234d_42b4_a30a_891581247013.slice/crio-b3fc52509693f706510dd37246d3b2f70fc10f18205c7187c68faf6520633d37 WatchSource:0}: Error finding container b3fc52509693f706510dd37246d3b2f70fc10f18205c7187c68faf6520633d37: Status 404 returned error can't find the container with id b3fc52509693f706510dd37246d3b2f70fc10f18205c7187c68faf6520633d37
Sep 16 15:03:59.058740 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:59.058714    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"
Sep 16 15:03:59.058939 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:59.058760    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5" event={"ID":"f5e46881-e473-442a-aa09-9f40ba9c017f","Type":"ContainerDied","Data":"aba9f815f8d634bfd5640ff7b4119821ff002072b5322cdab9dfddc2be3e3dd6"}
Sep 16 15:03:59.058939 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:59.058804    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aba9f815f8d634bfd5640ff7b4119821ff002072b5322cdab9dfddc2be3e3dd6"
Sep 16 15:03:59.060448 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:59.060396    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh" event={"ID":"80e084c0-234d-42b4-a30a-891581247013","Type":"ContainerStarted","Data":"b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac"}
Sep 16 15:03:59.060448 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:59.060433    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh" event={"ID":"80e084c0-234d-42b4-a30a-891581247013","Type":"ContainerStarted","Data":"b3fc52509693f706510dd37246d3b2f70fc10f18205c7187c68faf6520633d37"}
Sep 16 15:03:59.080269 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:03:59.080228    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh" podStartSLOduration=2.080213336 podStartE2EDuration="2.080213336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:03:59.079478873 +0000 UTC m=+1716.233557161" watchObservedRunningTime="2026-09-16 15:03:59.080213336 +0000 UTC m=+1716.234291622"
Sep 16 15:04:06.198042 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:06.197996    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"]
Sep 16 15:04:06.201380 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:06.201352    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ba69d2-6g2q5"]
Sep 16 15:04:07.295266 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:07.295231    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e46881-e473-442a-aa09-9f40ba9c017f" path="/var/lib/kubelet/pods/f5e46881-e473-442a-aa09-9f40ba9c017f/volumes"
Sep 16 15:04:07.795291 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:07.795253    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf" podUID="00e706e0-4c29-4109-aa90-a764d78acefb" containerName="main" containerID="cri-o://eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0" gracePeriod=45
Sep 16 15:04:08.048439 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.048378    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 15:04:08.113718 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.113690    2558 generic.go:356] "Generic (PLEG): container finished" podID="00e706e0-4c29-4109-aa90-a764d78acefb" containerID="eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0" exitCode=0
Sep 16 15:04:08.113887 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.113735    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"
Sep 16 15:04:08.113887 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.113779    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf" event={"ID":"00e706e0-4c29-4109-aa90-a764d78acefb","Type":"ContainerDied","Data":"eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0"}
Sep 16 15:04:08.113887 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.113827    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf" event={"ID":"00e706e0-4c29-4109-aa90-a764d78acefb","Type":"ContainerDied","Data":"522429ab6527e5dc3b1938f485077c911d806f6bde7632c29879eb0adbe42a17"}
Sep 16 15:04:08.113887 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.113848    2558 scope.go:117] "RemoveContainer" containerID="eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0"
Sep 16 15:04:08.122619 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.122600    2558 scope.go:117] "RemoveContainer" containerID="eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0"
Sep 16 15:04:08.122880 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:04:08.122863    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0\": container with ID starting with eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0 not found: ID does not exist" containerID="eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0"
Sep 16 15:04:08.122930 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.122887    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0"} err="failed to get container status \"eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0\": rpc error: code = NotFound desc = could not find container \"eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0\": container with ID starting with eddee66bf28dae409f2140ac677c7cf1fbbb6f7e2c77e10faf11ccf378241ae0 not found: ID does not exist"
Sep 16 15:04:08.220757 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.220720    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e706e0-4c29-4109-aa90-a764d78acefb-tls-certs\") pod \"00e706e0-4c29-4109-aa90-a764d78acefb\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") "
Sep 16 15:04:08.220927 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.220784    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-model-cache\") pod \"00e706e0-4c29-4109-aa90-a764d78acefb\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") "
Sep 16 15:04:08.220976 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.220924    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-dshm\") pod \"00e706e0-4c29-4109-aa90-a764d78acefb\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") "
Sep 16 15:04:08.220976 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.220967    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-home\") pod \"00e706e0-4c29-4109-aa90-a764d78acefb\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") "
Sep 16 15:04:08.221097 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.221035    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lgnr\" (UniqueName: \"kubernetes.io/projected/00e706e0-4c29-4109-aa90-a764d78acefb-kube-api-access-2lgnr\") pod \"00e706e0-4c29-4109-aa90-a764d78acefb\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") "
Sep 16 15:04:08.221097 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.221034    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-model-cache" (OuterVolumeSpecName: "model-cache") pod "00e706e0-4c29-4109-aa90-a764d78acefb" (UID: "00e706e0-4c29-4109-aa90-a764d78acefb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:04:08.221097 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.221093    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-tmp-dir\") pod \"00e706e0-4c29-4109-aa90-a764d78acefb\" (UID: \"00e706e0-4c29-4109-aa90-a764d78acefb\") "
Sep 16 15:04:08.221290 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.221235    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-home" (OuterVolumeSpecName: "home") pod "00e706e0-4c29-4109-aa90-a764d78acefb" (UID: "00e706e0-4c29-4109-aa90-a764d78acefb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:04:08.221410 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.221386    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00e706e0-4c29-4109-aa90-a764d78acefb" (UID: "00e706e0-4c29-4109-aa90-a764d78acefb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:04:08.221502 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.221488    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:08.221558 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.221504    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:08.221558 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.221515    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:08.223089 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.223071    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-dshm" (OuterVolumeSpecName: "dshm") pod "00e706e0-4c29-4109-aa90-a764d78acefb" (UID: "00e706e0-4c29-4109-aa90-a764d78acefb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:04:08.223193 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.223121    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e706e0-4c29-4109-aa90-a764d78acefb-kube-api-access-2lgnr" (OuterVolumeSpecName: "kube-api-access-2lgnr") pod "00e706e0-4c29-4109-aa90-a764d78acefb" (UID: "00e706e0-4c29-4109-aa90-a764d78acefb"). InnerVolumeSpecName "kube-api-access-2lgnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:04:08.223240 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.223200    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e706e0-4c29-4109-aa90-a764d78acefb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00e706e0-4c29-4109-aa90-a764d78acefb" (UID: "00e706e0-4c29-4109-aa90-a764d78acefb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:04:08.240992 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.240969    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:04:08.241118 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.241003    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:04:08.253562 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.253534    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:04:08.322314 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.322283    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00e706e0-4c29-4109-aa90-a764d78acefb-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:08.322314 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.322308    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00e706e0-4c29-4109-aa90-a764d78acefb-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:08.322314 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.322317    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lgnr\" (UniqueName: \"kubernetes.io/projected/00e706e0-4c29-4109-aa90-a764d78acefb-kube-api-access-2lgnr\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:08.437861 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.437826    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"]
Sep 16 15:04:08.445430 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:08.445399    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/auth-test-model-d28bd5-kserve-c6764d95f-mlttf"]
Sep 16 15:04:09.048871 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.048809    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw" podUID="88d4f8a8-846e-4a37-937a-7a978e8f9e24" containerName="main" containerID="cri-o://5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7" gracePeriod=45
Sep 16 15:04:09.131147 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.131119    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:04:09.296416 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.296379    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e706e0-4c29-4109-aa90-a764d78acefb" path="/var/lib/kubelet/pods/00e706e0-4c29-4109-aa90-a764d78acefb/volumes"
Sep 16 15:04:09.302361 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.302311    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 15:04:09.430951 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.430900    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-model-cache\") pod \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") "
Sep 16 15:04:09.430951 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.430955    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tmp-dir\") pod \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") "
Sep 16 15:04:09.431496 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.431002    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-home\") pod \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") "
Sep 16 15:04:09.431496 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.431036    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2p4m\" (UniqueName: \"kubernetes.io/projected/88d4f8a8-846e-4a37-937a-7a978e8f9e24-kube-api-access-d2p4m\") pod \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") "
Sep 16 15:04:09.431496 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.431068    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-dshm\") pod \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") "
Sep 16 15:04:09.431496 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.431086    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tls-certs\") pod \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\" (UID: \"88d4f8a8-846e-4a37-937a-7a978e8f9e24\") "
Sep 16 15:04:09.431496 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.431253    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-model-cache" (OuterVolumeSpecName: "model-cache") pod "88d4f8a8-846e-4a37-937a-7a978e8f9e24" (UID: "88d4f8a8-846e-4a37-937a-7a978e8f9e24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:04:09.431496 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.431357    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-home" (OuterVolumeSpecName: "home") pod "88d4f8a8-846e-4a37-937a-7a978e8f9e24" (UID: "88d4f8a8-846e-4a37-937a-7a978e8f9e24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:04:09.431496 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.431374    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88d4f8a8-846e-4a37-937a-7a978e8f9e24" (UID: "88d4f8a8-846e-4a37-937a-7a978e8f9e24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:04:09.431724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.431535    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:09.431724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.431548    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:09.431724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.431558    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:09.433319 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.433298    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-dshm" (OuterVolumeSpecName: "dshm") pod "88d4f8a8-846e-4a37-937a-7a978e8f9e24" (UID: "88d4f8a8-846e-4a37-937a-7a978e8f9e24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:04:09.433413 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.433383    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88d4f8a8-846e-4a37-937a-7a978e8f9e24" (UID: "88d4f8a8-846e-4a37-937a-7a978e8f9e24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:04:09.433474 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.433408    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d4f8a8-846e-4a37-937a-7a978e8f9e24-kube-api-access-d2p4m" (OuterVolumeSpecName: "kube-api-access-d2p4m") pod "88d4f8a8-846e-4a37-937a-7a978e8f9e24" (UID: "88d4f8a8-846e-4a37-937a-7a978e8f9e24"). InnerVolumeSpecName "kube-api-access-d2p4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:04:09.532404 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.532372    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2p4m\" (UniqueName: \"kubernetes.io/projected/88d4f8a8-846e-4a37-937a-7a978e8f9e24-kube-api-access-d2p4m\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:09.532404 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.532398    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d4f8a8-846e-4a37-937a-7a978e8f9e24-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:09.532404 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:09.532407    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d4f8a8-846e-4a37-937a-7a978e8f9e24-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:04:10.126424 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:10.126392    2558 generic.go:356] "Generic (PLEG): container finished" podID="88d4f8a8-846e-4a37-937a-7a978e8f9e24" containerID="5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7" exitCode=0
Sep 16 15:04:10.126642 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:10.126433    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"
Sep 16 15:04:10.126642 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:10.126484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw" event={"ID":"88d4f8a8-846e-4a37-937a-7a978e8f9e24","Type":"ContainerDied","Data":"5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7"}
Sep 16 15:04:10.126642 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:10.126532    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw" event={"ID":"88d4f8a8-846e-4a37-937a-7a978e8f9e24","Type":"ContainerDied","Data":"99a93859c9fd232ceccf9757bc37961d24a80e4943cd40833b3ab10a9ee48e08"}
Sep 16 15:04:10.126642 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:10.126558    2558 scope.go:117] "RemoveContainer" containerID="5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7"
Sep 16 15:04:10.135566 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:10.135546    2558 scope.go:117] "RemoveContainer" containerID="5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7"
Sep 16 15:04:10.135857 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:04:10.135838    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7\": container with ID starting with 5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7 not found: ID does not exist" containerID="5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7"
Sep 16 15:04:10.135901 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:10.135865    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7"} err="failed to get container status \"5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7\": rpc error: code = NotFound desc = could not find container \"5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7\": container with ID starting with 5b9491022ce38fa7b49ee0e93cf8f9839bb791f96ade29d4adc6f3f8ec9692d7 not found: ID does not exist"
Sep 16 15:04:10.148833 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:10.148809    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"]
Sep 16 15:04:10.153774 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:10.153754    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/auth-test-model-cadcf4-kserve-8476756df9-ch2hw"]
Sep 16 15:04:11.301077 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:04:11.301047    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d4f8a8-846e-4a37-937a-7a978e8f9e24" path="/var/lib/kubelet/pods/88d4f8a8-846e-4a37-937a-7a978e8f9e24/volumes"
Sep 16 15:05:13.623746 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:13.623712    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"]
Sep 16 15:05:23.332747 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:23.332717    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 15:05:23.337859 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:23.337837    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 15:05:23.641954 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:23.641874    2558 scope.go:117] "RemoveContainer" containerID="dabc262e561161571f5e1bd3f852ca8b978418f5773a2570eb382d3ce10b6268"
Sep 16 15:05:28.642248 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:28.642212    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh" podUID="80e084c0-234d-42b4-a30a-891581247013" containerName="main" containerID="cri-o://b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac" gracePeriod=45
Sep 16 15:05:28.894304 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:28.894248    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:05:29.048161 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.048123    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e084c0-234d-42b4-a30a-891581247013-tls-certs\") pod \"80e084c0-234d-42b4-a30a-891581247013\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") "
Sep 16 15:05:29.048367 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.048209    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccb8p\" (UniqueName: \"kubernetes.io/projected/80e084c0-234d-42b4-a30a-891581247013-kube-api-access-ccb8p\") pod \"80e084c0-234d-42b4-a30a-891581247013\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") "
Sep 16 15:05:29.048367 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.048265    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-home\") pod \"80e084c0-234d-42b4-a30a-891581247013\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") "
Sep 16 15:05:29.048367 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.048293    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-model-cache\") pod \"80e084c0-234d-42b4-a30a-891581247013\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") "
Sep 16 15:05:29.048367 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.048328    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-dshm\") pod \"80e084c0-234d-42b4-a30a-891581247013\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") "
Sep 16 15:05:29.048367 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.048358    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-tmp-dir\") pod \"80e084c0-234d-42b4-a30a-891581247013\" (UID: \"80e084c0-234d-42b4-a30a-891581247013\") "
Sep 16 15:05:29.048618 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.048587    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-model-cache" (OuterVolumeSpecName: "model-cache") pod "80e084c0-234d-42b4-a30a-891581247013" (UID: "80e084c0-234d-42b4-a30a-891581247013"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:05:29.048681 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.048598    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-home" (OuterVolumeSpecName: "home") pod "80e084c0-234d-42b4-a30a-891581247013" (UID: "80e084c0-234d-42b4-a30a-891581247013"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:05:29.048736 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.048694    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80e084c0-234d-42b4-a30a-891581247013" (UID: "80e084c0-234d-42b4-a30a-891581247013"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:05:29.048782 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.048733    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:05:29.048782 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.048746    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:05:29.050420 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.050396    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-dshm" (OuterVolumeSpecName: "dshm") pod "80e084c0-234d-42b4-a30a-891581247013" (UID: "80e084c0-234d-42b4-a30a-891581247013"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:05:29.050514 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.050399    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e084c0-234d-42b4-a30a-891581247013-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80e084c0-234d-42b4-a30a-891581247013" (UID: "80e084c0-234d-42b4-a30a-891581247013"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:05:29.050514 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.050446    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e084c0-234d-42b4-a30a-891581247013-kube-api-access-ccb8p" (OuterVolumeSpecName: "kube-api-access-ccb8p") pod "80e084c0-234d-42b4-a30a-891581247013" (UID: "80e084c0-234d-42b4-a30a-891581247013"). InnerVolumeSpecName "kube-api-access-ccb8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:05:29.149587 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.149493    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e084c0-234d-42b4-a30a-891581247013-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:05:29.149587 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.149528    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccb8p\" (UniqueName: \"kubernetes.io/projected/80e084c0-234d-42b4-a30a-891581247013-kube-api-access-ccb8p\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:05:29.149587 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.149542    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:05:29.149587 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.149553    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e084c0-234d-42b4-a30a-891581247013-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:05:29.534987 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.534905    2558 generic.go:356] "Generic (PLEG): container finished" podID="80e084c0-234d-42b4-a30a-891581247013" containerID="b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac" exitCode=0
Sep 16 15:05:29.534987 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.534947    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"
Sep 16 15:05:29.535223 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.534992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh" event={"ID":"80e084c0-234d-42b4-a30a-891581247013","Type":"ContainerDied","Data":"b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac"}
Sep 16 15:05:29.535223 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.535043    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh" event={"ID":"80e084c0-234d-42b4-a30a-891581247013","Type":"ContainerDied","Data":"b3fc52509693f706510dd37246d3b2f70fc10f18205c7187c68faf6520633d37"}
Sep 16 15:05:29.535223 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.535061    2558 scope.go:117] "RemoveContainer" containerID="b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac"
Sep 16 15:05:29.544054 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.544030    2558 scope.go:117] "RemoveContainer" containerID="b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac"
Sep 16 15:05:29.544369 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:05:29.544349    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac\": container with ID starting with b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac not found: ID does not exist" containerID="b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac"
Sep 16 15:05:29.544431 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.544381    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac"} err="failed to get container status \"b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac\": rpc error: code = NotFound desc = could not find container \"b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac\": container with ID starting with b798c167eaf871f571225ed2364c3bfb66df6c55f432eae1fa6a6d30248975ac not found: ID does not exist"
Sep 16 15:05:29.569594 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.569564    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"]
Sep 16 15:05:29.583714 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:29.583691    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/sub-test-model-d28bd5-kserve-74b647b949-gljsh"]
Sep 16 15:05:31.294670 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.294637    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e084c0-234d-42b4-a30a-891581247013" path="/var/lib/kubelet/pods/80e084c0-234d-42b4-a30a-891581247013/volumes"
Sep 16 15:05:31.817838 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.817809    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"]
Sep 16 15:05:31.818193 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818181    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e706e0-4c29-4109-aa90-a764d78acefb" containerName="main"
Sep 16 15:05:31.818248 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818195    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e706e0-4c29-4109-aa90-a764d78acefb" containerName="main"
Sep 16 15:05:31.818248 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818207    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d4f8a8-846e-4a37-937a-7a978e8f9e24" containerName="main"
Sep 16 15:05:31.818248 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818213    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d4f8a8-846e-4a37-937a-7a978e8f9e24" containerName="main"
Sep 16 15:05:31.818248 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818219    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e084c0-234d-42b4-a30a-891581247013" containerName="main"
Sep 16 15:05:31.818248 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818224    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e084c0-234d-42b4-a30a-891581247013" containerName="main"
Sep 16 15:05:31.818248 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818236    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e46881-e473-442a-aa09-9f40ba9c017f" containerName="revoke-keys"
Sep 16 15:05:31.818248 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818241    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e46881-e473-442a-aa09-9f40ba9c017f" containerName="revoke-keys"
Sep 16 15:05:31.818510 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818303    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5e46881-e473-442a-aa09-9f40ba9c017f" containerName="revoke-keys"
Sep 16 15:05:31.818510 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818311    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e084c0-234d-42b4-a30a-891581247013" containerName="main"
Sep 16 15:05:31.818510 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818316    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e706e0-4c29-4109-aa90-a764d78acefb" containerName="main"
Sep 16 15:05:31.818510 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.818322    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d4f8a8-846e-4a37-937a-7a978e8f9e24" containerName="main"
Sep 16 15:05:31.822576 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.822559    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:31.826495 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.826471    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d28bd5\"/\"default-dockercfg-m47zc\""
Sep 16 15:05:31.826807 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.826768    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d28bd5\"/\"openshift-service-ca.crt\""
Sep 16 15:05:31.826807 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.826772    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d28bd5\"/\"kube-root-ca.crt\""
Sep 16 15:05:31.826980 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.826898    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d28bd5\"/\"rate-test-model-d28bd5-kserve-self-signed-certs\""
Sep 16 15:05:31.828885 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.828866    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"]
Sep 16 15:05:31.974038 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.974000    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj2nb\" (UniqueName: \"kubernetes.io/projected/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-kube-api-access-hj2nb\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:31.974038 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.974043    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tls-certs\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:31.974327 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.974140    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tmp-dir\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:31.974327 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.974214    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-model-cache\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:31.974327 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.974246    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-home\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:31.974327 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:31.974261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-dshm\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.075506 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.075417    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tmp-dir\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.075506 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.075452    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-model-cache\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.075506 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.075470    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-home\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.075506 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.075485    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-dshm\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.075780 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.075531    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj2nb\" (UniqueName: \"kubernetes.io/projected/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-kube-api-access-hj2nb\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.075780 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.075559    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tls-certs\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.075916 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.075890    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tmp-dir\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.075979 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.075921    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-home\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.075979 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.075948    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-model-cache\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.077927 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.077896    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-dshm\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.078113 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.078099    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tls-certs\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.083680 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.083658    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj2nb\" (UniqueName: \"kubernetes.io/projected/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-kube-api-access-hj2nb\") pod \"rate-test-model-d28bd5-kserve-6487f5c799-sznl4\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") " pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.133502 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.133470    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:32.405492 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.405465    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"]
Sep 16 15:05:32.407024 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:05:32.406996    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e79f4e2_6d8f_4dc1_a453_a5bd56a98f9e.slice/crio-abf8b50a34d4a123a65f389be8a2f568c11363142f02871820153d0301fb2e7f WatchSource:0}: Error finding container abf8b50a34d4a123a65f389be8a2f568c11363142f02871820153d0301fb2e7f: Status 404 returned error can't find the container with id abf8b50a34d4a123a65f389be8a2f568c11363142f02871820153d0301fb2e7f
Sep 16 15:05:32.553536 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.553499    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4" event={"ID":"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e","Type":"ContainerStarted","Data":"7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e"}
Sep 16 15:05:32.553704 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.553545    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4" event={"ID":"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e","Type":"ContainerStarted","Data":"abf8b50a34d4a123a65f389be8a2f568c11363142f02871820153d0301fb2e7f"}
Sep 16 15:05:32.573362 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:32.573309    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4" podStartSLOduration=1.573293439 podStartE2EDuration="1.573293439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:05:32.571971737 +0000 UTC m=+1809.726050024" watchObservedRunningTime="2026-09-16 15:05:32.573293439 +0000 UTC m=+1809.727371726"
Sep 16 15:05:42.133856 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:42.133814    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:42.134282 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:42.134108    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:42.146631 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:42.146607    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:42.614496 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:42.614467    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:05:52.818958 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.818923    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"]
Sep 16 15:05:52.822450 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.822433    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.825340 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.825322    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cadcf4\"/\"kube-root-ca.crt\""
Sep 16 15:05:52.826448 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.826430    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cadcf4\"/\"openshift-service-ca.crt\""
Sep 16 15:05:52.826448 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.826436    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cadcf4\"/\"default-dockercfg-r6t6g\""
Sep 16 15:05:52.826582 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.826435    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cadcf4\"/\"rate-test-model-cadcf4-kserve-self-signed-certs\""
Sep 16 15:05:52.829712 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.829687    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"]
Sep 16 15:05:52.874043 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.874004    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-home\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.874239 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.874062    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-model-cache\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.874239 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.874085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274ac3e2-11ab-4250-b11b-00194e915d6a-tls-certs\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.874239 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.874104    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-dshm\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.874239 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.874140    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kc47\" (UniqueName: \"kubernetes.io/projected/274ac3e2-11ab-4250-b11b-00194e915d6a-kube-api-access-7kc47\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.874411 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.874242    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-tmp-dir\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.975235 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.975197    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-model-cache\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.975235 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.975237    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274ac3e2-11ab-4250-b11b-00194e915d6a-tls-certs\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.975495 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.975254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-dshm\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.975495 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.975282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kc47\" (UniqueName: \"kubernetes.io/projected/274ac3e2-11ab-4250-b11b-00194e915d6a-kube-api-access-7kc47\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.975495 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.975413    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-tmp-dir\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.975667 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.975546    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-home\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.975667 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.975659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-model-cache\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.975766 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.975746    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-tmp-dir\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.975844 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.975824    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-home\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.977625 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.977605    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-dshm\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.977907 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.977887    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274ac3e2-11ab-4250-b11b-00194e915d6a-tls-certs\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:52.983285 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:52.983263    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kc47\" (UniqueName: \"kubernetes.io/projected/274ac3e2-11ab-4250-b11b-00194e915d6a-kube-api-access-7kc47\") pod \"rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") " pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:53.133848 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:53.133756    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:05:53.404426 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:53.404402    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"]
Sep 16 15:05:53.406261 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:05:53.406235    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod274ac3e2_11ab_4250_b11b_00194e915d6a.slice/crio-7a18281ffc54f3fd911a536f99fef4a257362b5ca9919362887860bc4209c3c9 WatchSource:0}: Error finding container 7a18281ffc54f3fd911a536f99fef4a257362b5ca9919362887860bc4209c3c9: Status 404 returned error can't find the container with id 7a18281ffc54f3fd911a536f99fef4a257362b5ca9919362887860bc4209c3c9
Sep 16 15:05:53.668088 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:53.667992    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f" event={"ID":"274ac3e2-11ab-4250-b11b-00194e915d6a","Type":"ContainerStarted","Data":"a6e76473c44375ea076e5be0f759b2d8aafab5cf45b8f27732be418f64d60ca5"}
Sep 16 15:05:53.668088 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:53.668043    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f" event={"ID":"274ac3e2-11ab-4250-b11b-00194e915d6a","Type":"ContainerStarted","Data":"7a18281ffc54f3fd911a536f99fef4a257362b5ca9919362887860bc4209c3c9"}
Sep 16 15:05:53.687463 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:05:53.687408    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f" podStartSLOduration=1.68739172 podStartE2EDuration="1.68739172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:05:53.685276338 +0000 UTC m=+1830.839354626" watchObservedRunningTime="2026-09-16 15:05:53.68739172 +0000 UTC m=+1830.841470008"
Sep 16 15:06:03.134213 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:03.134184    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:06:03.136659 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:03.134246    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:06:03.146539 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:03.146516    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:06:03.735503 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:03.735474    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:06:23.667556 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:23.664182    2558 scope.go:117] "RemoveContainer" containerID="2c8303af4f4864e17342f27229bbd9a2776d46de859d502e206db974800f48df"
Sep 16 15:06:23.676147 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:23.676128    2558 scope.go:117] "RemoveContainer" containerID="4f5675d869daa4b0a03aaf0c39d5c7f9b049985ab23a64b42e7534091154fb2a"
Sep 16 15:06:23.684071 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:23.684050    2558 scope.go:117] "RemoveContainer" containerID="0cedf449322f52f089005d9f0b9d1c72164b7e025848f5d13e043d93ca5374fc"
Sep 16 15:06:38.887824 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:38.887793    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"]
Sep 16 15:06:39.826435 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:39.826406    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"]
Sep 16 15:06:45.961552 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:45.961476    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"]
Sep 16 15:06:45.965011 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:45.964994    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:06:45.967744 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:45.967722    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-91f4ab\""
Sep 16 15:06:45.967858 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:45.967724    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-91f4ab-dockercfg-bzks6\""
Sep 16 15:06:45.975830 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:45.975808    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"]
Sep 16 15:06:46.059397 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.059360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfpw\" (UniqueName: \"kubernetes.io/projected/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-kube-api-access-bjfpw\") pod \"payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd\" (UID: \"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:06:46.059643 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.059621    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd\" (UID: \"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:06:46.160879 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.160841    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfpw\" (UniqueName: \"kubernetes.io/projected/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-kube-api-access-bjfpw\") pod \"payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd\" (UID: \"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:06:46.161077 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.160892    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd\" (UID: \"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:06:46.161617 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.161596    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd\" (UID: \"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:06:46.169578 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.169560    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfpw\" (UniqueName: \"kubernetes.io/projected/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-kube-api-access-bjfpw\") pod \"payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd\" (UID: \"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:06:46.275836 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.275752    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:06:46.549958 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.549934    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"]
Sep 16 15:06:46.552409 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:06:46.552377    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8d8b9b_4471_46a1_ad34_1eed71fbe26d.slice/crio-e9bd31b8b2fc7ffd787e1d3029b640e351af9bb5c4d564bc7a1c888b4edf0cba WatchSource:0}: Error finding container e9bd31b8b2fc7ffd787e1d3029b640e351af9bb5c4d564bc7a1c888b4edf0cba: Status 404 returned error can't find the container with id e9bd31b8b2fc7ffd787e1d3029b640e351af9bb5c4d564bc7a1c888b4edf0cba
Sep 16 15:06:46.951451 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.951418    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd" event={"ID":"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d","Type":"ContainerStarted","Data":"e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95"}
Sep 16 15:06:46.951648 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.951465    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd" event={"ID":"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d","Type":"ContainerStarted","Data":"e9bd31b8b2fc7ffd787e1d3029b640e351af9bb5c4d564bc7a1c888b4edf0cba"}
Sep 16 15:06:46.951648 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.951555    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:06:46.968227 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:46.968158    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd" podStartSLOduration=1.9681407279999998 podStartE2EDuration="1.968140728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:06:46.96724791 +0000 UTC m=+1884.121326200" watchObservedRunningTime="2026-09-16 15:06:46.968140728 +0000 UTC m=+1884.122219021"
Sep 16 15:06:53.906180 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:53.906127    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4" podUID="0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" containerName="main" containerID="cri-o://7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e" gracePeriod=45
Sep 16 15:06:54.151596 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.151569    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:06:54.239774 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.239692    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-model-cache\") pod \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") "
Sep 16 15:06:54.239774 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.239742    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tls-certs\") pod \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") "
Sep 16 15:06:54.240008 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.239785    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj2nb\" (UniqueName: \"kubernetes.io/projected/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-kube-api-access-hj2nb\") pod \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") "
Sep 16 15:06:54.240008 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.239820    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-dshm\") pod \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") "
Sep 16 15:06:54.240008 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.239838    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tmp-dir\") pod \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") "
Sep 16 15:06:54.240008 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.239857    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-home\") pod \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\" (UID: \"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e\") "
Sep 16 15:06:54.240008 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.239967    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" (UID: "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:06:54.240345 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.240213    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" (UID: "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:06:54.240345 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.240238    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-home" (OuterVolumeSpecName: "home") pod "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" (UID: "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:06:54.240345 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.240240    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:54.241835 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.241811    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-dshm" (OuterVolumeSpecName: "dshm") pod "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" (UID: "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:06:54.241973 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.241937    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" (UID: "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:06:54.242024 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.241971    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-kube-api-access-hj2nb" (OuterVolumeSpecName: "kube-api-access-hj2nb") pod "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" (UID: "0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e"). InnerVolumeSpecName "kube-api-access-hj2nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:06:54.341586 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.341557    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:54.341586 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.341583    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj2nb\" (UniqueName: \"kubernetes.io/projected/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-kube-api-access-hj2nb\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:54.341586 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.341593    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:54.341806 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.341602    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:54.341806 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.341611    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:54.844746 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.844699    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f" podUID="274ac3e2-11ab-4250-b11b-00194e915d6a" containerName="main" containerID="cri-o://a6e76473c44375ea076e5be0f759b2d8aafab5cf45b8f27732be418f64d60ca5" gracePeriod=45
Sep 16 15:06:54.993041 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.993007    2558 generic.go:356] "Generic (PLEG): container finished" podID="0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" containerID="7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e" exitCode=0
Sep 16 15:06:54.993530 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.993058    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"
Sep 16 15:06:54.993530 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.993084    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4" event={"ID":"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e","Type":"ContainerDied","Data":"7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e"}
Sep 16 15:06:54.993530 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.993126    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4" event={"ID":"0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e","Type":"ContainerDied","Data":"abf8b50a34d4a123a65f389be8a2f568c11363142f02871820153d0301fb2e7f"}
Sep 16 15:06:54.993530 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.993144    2558 scope.go:117] "RemoveContainer" containerID="7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e"
Sep 16 15:06:54.998486 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.998212    2558 generic.go:356] "Generic (PLEG): container finished" podID="274ac3e2-11ab-4250-b11b-00194e915d6a" containerID="a6e76473c44375ea076e5be0f759b2d8aafab5cf45b8f27732be418f64d60ca5" exitCode=0
Sep 16 15:06:54.998486 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:54.998293    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f" event={"ID":"274ac3e2-11ab-4250-b11b-00194e915d6a","Type":"ContainerDied","Data":"a6e76473c44375ea076e5be0f759b2d8aafab5cf45b8f27732be418f64d60ca5"}
Sep 16 15:06:55.004773 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.004756    2558 scope.go:117] "RemoveContainer" containerID="7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e"
Sep 16 15:06:55.005063 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:06:55.005044    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e\": container with ID starting with 7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e not found: ID does not exist" containerID="7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e"
Sep 16 15:06:55.005112 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.005072    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e"} err="failed to get container status \"7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e\": rpc error: code = NotFound desc = could not find container \"7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e\": container with ID starting with 7ba7febef2fc94a13552e5bce51bdb0a91496fb42da73db75810c73480de904e not found: ID does not exist"
Sep 16 15:06:55.019070 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.019046    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"]
Sep 16 15:06:55.022620 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.022601    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d28bd5/rate-test-model-d28bd5-kserve-6487f5c799-sznl4"]
Sep 16 15:06:55.107472 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.107423    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:06:55.251733 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.251702    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kc47\" (UniqueName: \"kubernetes.io/projected/274ac3e2-11ab-4250-b11b-00194e915d6a-kube-api-access-7kc47\") pod \"274ac3e2-11ab-4250-b11b-00194e915d6a\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") "
Sep 16 15:06:55.251899 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.251746    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-model-cache\") pod \"274ac3e2-11ab-4250-b11b-00194e915d6a\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") "
Sep 16 15:06:55.251899 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.251778    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-home\") pod \"274ac3e2-11ab-4250-b11b-00194e915d6a\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") "
Sep 16 15:06:55.251899 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.251824    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274ac3e2-11ab-4250-b11b-00194e915d6a-tls-certs\") pod \"274ac3e2-11ab-4250-b11b-00194e915d6a\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") "
Sep 16 15:06:55.251899 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.251857    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-tmp-dir\") pod \"274ac3e2-11ab-4250-b11b-00194e915d6a\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") "
Sep 16 15:06:55.252066 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.251944    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-dshm\") pod \"274ac3e2-11ab-4250-b11b-00194e915d6a\" (UID: \"274ac3e2-11ab-4250-b11b-00194e915d6a\") "
Sep 16 15:06:55.252066 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.252010    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "274ac3e2-11ab-4250-b11b-00194e915d6a" (UID: "274ac3e2-11ab-4250-b11b-00194e915d6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:06:55.252208 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.252086    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-home" (OuterVolumeSpecName: "home") pod "274ac3e2-11ab-4250-b11b-00194e915d6a" (UID: "274ac3e2-11ab-4250-b11b-00194e915d6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:06:55.252208 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.252123    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "274ac3e2-11ab-4250-b11b-00194e915d6a" (UID: "274ac3e2-11ab-4250-b11b-00194e915d6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:06:55.252323 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.252245    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:55.252323 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.252258    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:55.252323 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.252285    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:55.253914 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.253885    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-dshm" (OuterVolumeSpecName: "dshm") pod "274ac3e2-11ab-4250-b11b-00194e915d6a" (UID: "274ac3e2-11ab-4250-b11b-00194e915d6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:06:55.254024 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.253912    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274ac3e2-11ab-4250-b11b-00194e915d6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "274ac3e2-11ab-4250-b11b-00194e915d6a" (UID: "274ac3e2-11ab-4250-b11b-00194e915d6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:06:55.254024 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.253976    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274ac3e2-11ab-4250-b11b-00194e915d6a-kube-api-access-7kc47" (OuterVolumeSpecName: "kube-api-access-7kc47") pod "274ac3e2-11ab-4250-b11b-00194e915d6a" (UID: "274ac3e2-11ab-4250-b11b-00194e915d6a"). InnerVolumeSpecName "kube-api-access-7kc47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:06:55.294775 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.294749    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" path="/var/lib/kubelet/pods/0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e/volumes"
Sep 16 15:06:55.353425 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.353400    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274ac3e2-11ab-4250-b11b-00194e915d6a-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:55.353425 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.353419    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274ac3e2-11ab-4250-b11b-00194e915d6a-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:55.353425 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:55.353428    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kc47\" (UniqueName: \"kubernetes.io/projected/274ac3e2-11ab-4250-b11b-00194e915d6a-kube-api-access-7kc47\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:06:56.004681 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:56.004642    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"
Sep 16 15:06:56.005132 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:56.004677    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f" event={"ID":"274ac3e2-11ab-4250-b11b-00194e915d6a","Type":"ContainerDied","Data":"7a18281ffc54f3fd911a536f99fef4a257362b5ca9919362887860bc4209c3c9"}
Sep 16 15:06:56.005132 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:56.004725    2558 scope.go:117] "RemoveContainer" containerID="a6e76473c44375ea076e5be0f759b2d8aafab5cf45b8f27732be418f64d60ca5"
Sep 16 15:06:56.022033 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:56.022012    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"]
Sep 16 15:06:56.025115 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:56.025095    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cadcf4/rate-test-model-cadcf4-kserve-5dc6bd5487-55z6f"]
Sep 16 15:06:57.295384 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:57.295352    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="274ac3e2-11ab-4250-b11b-00194e915d6a" path="/var/lib/kubelet/pods/274ac3e2-11ab-4250-b11b-00194e915d6a/volumes"
Sep 16 15:06:57.960725 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:06:57.960694    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:07:15.912125 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.912095    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"]
Sep 16 15:07:15.912586 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.912467    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="274ac3e2-11ab-4250-b11b-00194e915d6a" containerName="main"
Sep 16 15:07:15.912586 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.912479    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="274ac3e2-11ab-4250-b11b-00194e915d6a" containerName="main"
Sep 16 15:07:15.912586 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.912505    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" containerName="main"
Sep 16 15:07:15.912586 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.912510    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" containerName="main"
Sep 16 15:07:15.912586 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.912561    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="274ac3e2-11ab-4250-b11b-00194e915d6a" containerName="main"
Sep 16 15:07:15.912586 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.912572    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e79f4e2-6d8f-4dc1-a453-a5bd56a98f9e" containerName="main"
Sep 16 15:07:15.915914 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.915894    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:07:15.919714 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.919690    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-438a9b-dockercfg-ff44d\""
Sep 16 15:07:15.919999 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.919974    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-438a9b\""
Sep 16 15:07:15.929122 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.929100    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"]
Sep 16 15:07:15.932516 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.932496    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"]
Sep 16 15:07:15.932607 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.932579    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:07:15.941969 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.941946    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"]
Sep 16 15:07:15.942387 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.942359    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl\" (UID: \"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:07:15.942523 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:15.942510    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rdp\" (UniqueName: \"kubernetes.io/projected/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-kube-api-access-n4rdp\") pod \"payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl\" (UID: \"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:07:16.043860 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.043828    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c3dffb-4097-4373-9119-99aa4a01577b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6\" (UID: \"36c3dffb-4097-4373-9119-99aa4a01577b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:07:16.044039 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.043878    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvzfq\" (UniqueName: \"kubernetes.io/projected/36c3dffb-4097-4373-9119-99aa4a01577b-kube-api-access-bvzfq\") pod \"payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6\" (UID: \"36c3dffb-4097-4373-9119-99aa4a01577b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:07:16.044039 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.043911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rdp\" (UniqueName: \"kubernetes.io/projected/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-kube-api-access-n4rdp\") pod \"payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl\" (UID: \"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:07:16.044039 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.043947    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl\" (UID: \"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:07:16.044720 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.044702    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl\" (UID: \"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:07:16.052262 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.052238    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rdp\" (UniqueName: \"kubernetes.io/projected/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-kube-api-access-n4rdp\") pod \"payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl\" (UID: \"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:07:16.144909 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.144875    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c3dffb-4097-4373-9119-99aa4a01577b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6\" (UID: \"36c3dffb-4097-4373-9119-99aa4a01577b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:07:16.145102 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.144942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvzfq\" (UniqueName: \"kubernetes.io/projected/36c3dffb-4097-4373-9119-99aa4a01577b-kube-api-access-bvzfq\") pod \"payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6\" (UID: \"36c3dffb-4097-4373-9119-99aa4a01577b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:07:16.145568 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.145550    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c3dffb-4097-4373-9119-99aa4a01577b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6\" (UID: \"36c3dffb-4097-4373-9119-99aa4a01577b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:07:16.153093 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.153060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvzfq\" (UniqueName: \"kubernetes.io/projected/36c3dffb-4097-4373-9119-99aa4a01577b-kube-api-access-bvzfq\") pod \"payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6\" (UID: \"36c3dffb-4097-4373-9119-99aa4a01577b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:07:16.226979 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.226893    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:07:16.243461 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.243379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:07:16.504315 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.504235    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"]
Sep 16 15:07:16.507482 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:07:16.507451    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd801944_eefe_47e5_9ae1_9cb1c9b93a6a.slice/crio-79b252041d1d79b3470039a8196e50486792f653b12ba31a3e90b838fa90f847 WatchSource:0}: Error finding container 79b252041d1d79b3470039a8196e50486792f653b12ba31a3e90b838fa90f847: Status 404 returned error can't find the container with id 79b252041d1d79b3470039a8196e50486792f653b12ba31a3e90b838fa90f847
Sep 16 15:07:16.531433 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:16.531408    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"]
Sep 16 15:07:16.544364 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:07:16.544340    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c3dffb_4097_4373_9119_99aa4a01577b.slice/crio-800bf64f703de06202e258ebc6ea48d2a88dfd0ec38a0e27dd7053001620b51c WatchSource:0}: Error finding container 800bf64f703de06202e258ebc6ea48d2a88dfd0ec38a0e27dd7053001620b51c: Status 404 returned error can't find the container with id 800bf64f703de06202e258ebc6ea48d2a88dfd0ec38a0e27dd7053001620b51c
Sep 16 15:07:17.114286 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:17.114245    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6" event={"ID":"36c3dffb-4097-4373-9119-99aa4a01577b","Type":"ContainerStarted","Data":"e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de"}
Sep 16 15:07:17.114756 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:17.114313    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6" event={"ID":"36c3dffb-4097-4373-9119-99aa4a01577b","Type":"ContainerStarted","Data":"800bf64f703de06202e258ebc6ea48d2a88dfd0ec38a0e27dd7053001620b51c"}
Sep 16 15:07:17.114756 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:17.114400    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:07:17.116023 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:17.115994    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl" event={"ID":"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a","Type":"ContainerStarted","Data":"9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6"}
Sep 16 15:07:17.116147 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:17.116039    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl" event={"ID":"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a","Type":"ContainerStarted","Data":"79b252041d1d79b3470039a8196e50486792f653b12ba31a3e90b838fa90f847"}
Sep 16 15:07:17.116147 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:17.116066    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:07:17.133784 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:17.133739    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6" podStartSLOduration=2.133726354 podStartE2EDuration="2.133726354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:07:17.132265406 +0000 UTC m=+1914.286343692" watchObservedRunningTime="2026-09-16 15:07:17.133726354 +0000 UTC m=+1914.287804690"
Sep 16 15:07:17.148980 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:17.148937    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl" podStartSLOduration=2.148925184 podStartE2EDuration="2.148925184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:07:17.147795081 +0000 UTC m=+1914.301873370" watchObservedRunningTime="2026-09-16 15:07:17.148925184 +0000 UTC m=+1914.303003471"
Sep 16 15:07:28.123350 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:28.123320    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:07:28.123838 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:28.123549    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:07:48.204995 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:48.204962    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4z9lb_4b0b07a0-1590-46ec-bc89-937aeeb72f6f/payload-processing/0.log"
Sep 16 15:07:48.321198 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:07:48.321154    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6_36c3dffb-4097-4373-9119-99aa4a01577b/payload-processing/0.log"
Sep 16 15:08:13.950648 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:13.950619    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd_1c8d8b9b-4471-46a1-ad34-1eed71fbe26d/payload-processing/0.log"
Sep 16 15:08:14.074419 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:14.074393    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4z9lb_4b0b07a0-1590-46ec-bc89-937aeeb72f6f/payload-processing/0.log"
Sep 16 15:08:14.182686 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:14.182656    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6_36c3dffb-4097-4373-9119-99aa4a01577b/payload-processing/0.log"
Sep 16 15:08:22.714666 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:22.714591    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"]
Sep 16 15:08:22.718417 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:22.718399    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:08:22.721393 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:22.721376    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5fdb6e-dockercfg-47swt\""
Sep 16 15:08:22.721500 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:22.721431    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5fdb6e\""
Sep 16 15:08:22.730050 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:22.730030    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"]
Sep 16 15:08:22.867700 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:22.867666    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k\" (UID: \"d738fbcd-47bd-4ea7-8d96-19f638aaeefe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:08:22.867888 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:22.867732    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbnkp\" (UniqueName: \"kubernetes.io/projected/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-kube-api-access-rbnkp\") pod \"payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k\" (UID: \"d738fbcd-47bd-4ea7-8d96-19f638aaeefe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:08:22.968821 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:22.968744    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k\" (UID: \"d738fbcd-47bd-4ea7-8d96-19f638aaeefe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:08:22.968821 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:22.968805    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbnkp\" (UniqueName: \"kubernetes.io/projected/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-kube-api-access-rbnkp\") pod \"payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k\" (UID: \"d738fbcd-47bd-4ea7-8d96-19f638aaeefe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:08:22.969465 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:22.969444    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k\" (UID: \"d738fbcd-47bd-4ea7-8d96-19f638aaeefe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:08:22.978210 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:22.978185    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbnkp\" (UniqueName: \"kubernetes.io/projected/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-kube-api-access-rbnkp\") pod \"payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k\" (UID: \"d738fbcd-47bd-4ea7-8d96-19f638aaeefe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:08:23.029064 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:23.029036    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:08:23.304736 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:23.304712    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"]
Sep 16 15:08:23.306378 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:08:23.306356    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd738fbcd_47bd_4ea7_8d96_19f638aaeefe.slice/crio-ffbbcf6ae1a59e1159e53f207b560f1ccc21e4e53316f6094f4524c87f723f6e WatchSource:0}: Error finding container ffbbcf6ae1a59e1159e53f207b560f1ccc21e4e53316f6094f4524c87f723f6e: Status 404 returned error can't find the container with id ffbbcf6ae1a59e1159e53f207b560f1ccc21e4e53316f6094f4524c87f723f6e
Sep 16 15:08:23.459939 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:23.459902    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k" event={"ID":"d738fbcd-47bd-4ea7-8d96-19f638aaeefe","Type":"ContainerStarted","Data":"a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545"}
Sep 16 15:08:23.460143 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:23.459975    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:08:23.460143 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:23.459996    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k" event={"ID":"d738fbcd-47bd-4ea7-8d96-19f638aaeefe","Type":"ContainerStarted","Data":"ffbbcf6ae1a59e1159e53f207b560f1ccc21e4e53316f6094f4524c87f723f6e"}
Sep 16 15:08:23.480508 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:23.480459    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k" podStartSLOduration=1.480444004 podStartE2EDuration="1.480444004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:08:23.477494915 +0000 UTC m=+1980.631573241" watchObservedRunningTime="2026-09-16 15:08:23.480444004 +0000 UTC m=+1980.634522291"
Sep 16 15:08:23.733754 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:23.733670    2558 scope.go:117] "RemoveContainer" containerID="0874aec5051d7cb2302c8853a13b66989cfa34dad05edece602d63d5fd109462"
Sep 16 15:08:34.467691 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:34.467663    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:08:48.571712 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:48.571678    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"]
Sep 16 15:08:48.575390 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:48.575363    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"
Sep 16 15:08:48.578437 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:48.578419    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:08:48.593671 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:48.593650    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"]
Sep 16 15:08:48.698921 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:48.698879    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95fd407-a4d0-43d0-8732-2f592db13c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z\" (UID: \"e95fd407-a4d0-43d0-8732-2f592db13c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"
Sep 16 15:08:48.800331 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:48.800293    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95fd407-a4d0-43d0-8732-2f592db13c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z\" (UID: \"e95fd407-a4d0-43d0-8732-2f592db13c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"
Sep 16 15:08:48.800879 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:48.800860    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95fd407-a4d0-43d0-8732-2f592db13c1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z\" (UID: \"e95fd407-a4d0-43d0-8732-2f592db13c1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"
Sep 16 15:08:48.885273 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:48.885193    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"
Sep 16 15:08:49.194443 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:49.194410    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"]
Sep 16 15:08:49.196117 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:08:49.196088    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95fd407_a4d0_43d0_8732_2f592db13c1d.slice/crio-5fbbee57572d099aa7f2f876b771440e9f65e6ea9bc238bf9c1a64496bddfbe8 WatchSource:0}: Error finding container 5fbbee57572d099aa7f2f876b771440e9f65e6ea9bc238bf9c1a64496bddfbe8: Status 404 returned error can't find the container with id 5fbbee57572d099aa7f2f876b771440e9f65e6ea9bc238bf9c1a64496bddfbe8
Sep 16 15:08:49.592823 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:49.592784    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z" event={"ID":"e95fd407-a4d0-43d0-8732-2f592db13c1d","Type":"ContainerStarted","Data":"91e709fc962059cc051e68c54d0447d2dae3a5387e955884df8b798f13fe5663"}
Sep 16 15:08:49.593302 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:49.592848    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z" event={"ID":"e95fd407-a4d0-43d0-8732-2f592db13c1d","Type":"ContainerStarted","Data":"5fbbee57572d099aa7f2f876b771440e9f65e6ea9bc238bf9c1a64496bddfbe8"}
Sep 16 15:08:49.620560 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:49.620512    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z" podStartSLOduration=1.620495362 podStartE2EDuration="1.620495362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:08:49.618617131 +0000 UTC m=+2006.772695418" watchObservedRunningTime="2026-09-16 15:08:49.620495362 +0000 UTC m=+2006.774573702"
Sep 16 15:08:50.599221 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:50.599189    2558 generic.go:356] "Generic (PLEG): container finished" podID="e95fd407-a4d0-43d0-8732-2f592db13c1d" containerID="91e709fc962059cc051e68c54d0447d2dae3a5387e955884df8b798f13fe5663" exitCode=0
Sep 16 15:08:50.599603 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:50.599266    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z" event={"ID":"e95fd407-a4d0-43d0-8732-2f592db13c1d","Type":"ContainerDied","Data":"91e709fc962059cc051e68c54d0447d2dae3a5387e955884df8b798f13fe5663"}
Sep 16 15:08:51.737175 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:51.737132    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"
Sep 16 15:08:51.826782 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:51.826750    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95fd407-a4d0-43d0-8732-2f592db13c1d-maas-api-service-ca\") pod \"e95fd407-a4d0-43d0-8732-2f592db13c1d\" (UID: \"e95fd407-a4d0-43d0-8732-2f592db13c1d\") "
Sep 16 15:08:51.827128 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:51.827105    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95fd407-a4d0-43d0-8732-2f592db13c1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e95fd407-a4d0-43d0-8732-2f592db13c1d" (UID: "e95fd407-a4d0-43d0-8732-2f592db13c1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:08:51.928206 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:51.928111    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e95fd407-a4d0-43d0-8732-2f592db13c1d-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:08:52.611554 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:52.611517    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z" event={"ID":"e95fd407-a4d0-43d0-8732-2f592db13c1d","Type":"ContainerDied","Data":"5fbbee57572d099aa7f2f876b771440e9f65e6ea9bc238bf9c1a64496bddfbe8"}
Sep 16 15:08:52.611554 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:52.611552    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"
Sep 16 15:08:52.611771 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:08:52.611592    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fbbee57572d099aa7f2f876b771440e9f65e6ea9bc238bf9c1a64496bddfbe8"
Sep 16 15:09:00.054177 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.054129    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"]
Sep 16 15:09:00.054683 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.054424    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k" podUID="d738fbcd-47bd-4ea7-8d96-19f638aaeefe" containerName="payload-processing" containerID="cri-o://a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545" gracePeriod=30
Sep 16 15:09:00.311610 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.311554    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:09:00.406092 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.406062    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbnkp\" (UniqueName: \"kubernetes.io/projected/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-kube-api-access-rbnkp\") pod \"d738fbcd-47bd-4ea7-8d96-19f638aaeefe\" (UID: \"d738fbcd-47bd-4ea7-8d96-19f638aaeefe\") "
Sep 16 15:09:00.406289 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.406153    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-plugins-config-volume\") pod \"d738fbcd-47bd-4ea7-8d96-19f638aaeefe\" (UID: \"d738fbcd-47bd-4ea7-8d96-19f638aaeefe\") "
Sep 16 15:09:00.406524 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.406500    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d738fbcd-47bd-4ea7-8d96-19f638aaeefe" (UID: "d738fbcd-47bd-4ea7-8d96-19f638aaeefe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:09:00.408174 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.408126    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-kube-api-access-rbnkp" (OuterVolumeSpecName: "kube-api-access-rbnkp") pod "d738fbcd-47bd-4ea7-8d96-19f638aaeefe" (UID: "d738fbcd-47bd-4ea7-8d96-19f638aaeefe"). InnerVolumeSpecName "kube-api-access-rbnkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:09:00.446600 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.446565    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"]
Sep 16 15:09:00.464396 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.464363    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5fdb6e-lc64z"]
Sep 16 15:09:00.507439 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.507410    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbnkp\" (UniqueName: \"kubernetes.io/projected/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-kube-api-access-rbnkp\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:09:00.507439 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.507435    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d738fbcd-47bd-4ea7-8d96-19f638aaeefe-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:09:00.657763 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.657673    2558 generic.go:356] "Generic (PLEG): container finished" podID="d738fbcd-47bd-4ea7-8d96-19f638aaeefe" containerID="a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545" exitCode=0
Sep 16 15:09:00.657763 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.657716    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"
Sep 16 15:09:00.657763 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.657742    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k" event={"ID":"d738fbcd-47bd-4ea7-8d96-19f638aaeefe","Type":"ContainerDied","Data":"a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545"}
Sep 16 15:09:00.658014 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.657786    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k" event={"ID":"d738fbcd-47bd-4ea7-8d96-19f638aaeefe","Type":"ContainerDied","Data":"ffbbcf6ae1a59e1159e53f207b560f1ccc21e4e53316f6094f4524c87f723f6e"}
Sep 16 15:09:00.658014 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.657809    2558 scope.go:117] "RemoveContainer" containerID="a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545"
Sep 16 15:09:00.668130 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.668111    2558 scope.go:117] "RemoveContainer" containerID="a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545"
Sep 16 15:09:00.668433 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:09:00.668413    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545\": container with ID starting with a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545 not found: ID does not exist" containerID="a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545"
Sep 16 15:09:00.668494 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.668440    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545"} err="failed to get container status \"a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545\": rpc error: code = NotFound desc = could not find container \"a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545\": container with ID starting with a165ba5b52d62d46b9a88936fedab695781eab109dec786c163a7a81d4dab545 not found: ID does not exist"
Sep 16 15:09:00.692470 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.692438    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"]
Sep 16 15:09:00.696891 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:00.696867    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fdb6e-848d8977ff-g9c2k"]
Sep 16 15:09:01.295500 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:01.295464    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d738fbcd-47bd-4ea7-8d96-19f638aaeefe" path="/var/lib/kubelet/pods/d738fbcd-47bd-4ea7-8d96-19f638aaeefe/volumes"
Sep 16 15:09:01.295884 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:01.295795    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e95fd407-a4d0-43d0-8732-2f592db13c1d" path="/var/lib/kubelet/pods/e95fd407-a4d0-43d0-8732-2f592db13c1d/volumes"
Sep 16 15:09:11.668848 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.668807    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"]
Sep 16 15:09:11.669449 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.669203    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e95fd407-a4d0-43d0-8732-2f592db13c1d" containerName="revoke-keys"
Sep 16 15:09:11.669449 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.669217    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e95fd407-a4d0-43d0-8732-2f592db13c1d" containerName="revoke-keys"
Sep 16 15:09:11.669449 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.669231    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d738fbcd-47bd-4ea7-8d96-19f638aaeefe" containerName="payload-processing"
Sep 16 15:09:11.669449 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.669236    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d738fbcd-47bd-4ea7-8d96-19f638aaeefe" containerName="payload-processing"
Sep 16 15:09:11.669449 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.669317    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e95fd407-a4d0-43d0-8732-2f592db13c1d" containerName="revoke-keys"
Sep 16 15:09:11.669449 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.669324    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d738fbcd-47bd-4ea7-8d96-19f638aaeefe" containerName="payload-processing"
Sep 16 15:09:11.673843 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.673818    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"
Sep 16 15:09:11.677095 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.677067    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:09:11.681492 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.681465    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"]
Sep 16 15:09:11.818265 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.818229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1814869e-f91b-4d8c-a56e-6328fc7bba88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z\" (UID: \"1814869e-f91b-4d8c-a56e-6328fc7bba88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"
Sep 16 15:09:11.919272 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.919156    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1814869e-f91b-4d8c-a56e-6328fc7bba88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z\" (UID: \"1814869e-f91b-4d8c-a56e-6328fc7bba88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"
Sep 16 15:09:11.919829 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.919804    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1814869e-f91b-4d8c-a56e-6328fc7bba88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z\" (UID: \"1814869e-f91b-4d8c-a56e-6328fc7bba88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"
Sep 16 15:09:11.986192 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:11.986134    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"
Sep 16 15:09:12.260005 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:12.259975    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"]
Sep 16 15:09:12.262027 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:09:12.262001    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1814869e_f91b_4d8c_a56e_6328fc7bba88.slice/crio-4c2b4f0ae3688f61145147190000aed2cc012292649f8c14b72937c199ac9ec7 WatchSource:0}: Error finding container 4c2b4f0ae3688f61145147190000aed2cc012292649f8c14b72937c199ac9ec7: Status 404 returned error can't find the container with id 4c2b4f0ae3688f61145147190000aed2cc012292649f8c14b72937c199ac9ec7
Sep 16 15:09:12.728282 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:12.728242    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z" event={"ID":"1814869e-f91b-4d8c-a56e-6328fc7bba88","Type":"ContainerStarted","Data":"ac27aa85902d2bf6fc91439b104c22f25acf1c517e67106af1ec084321ac3205"}
Sep 16 15:09:12.728709 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:12.728298    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z" event={"ID":"1814869e-f91b-4d8c-a56e-6328fc7bba88","Type":"ContainerStarted","Data":"4c2b4f0ae3688f61145147190000aed2cc012292649f8c14b72937c199ac9ec7"}
Sep 16 15:09:12.750844 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:12.750786    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z" podStartSLOduration=1.7507653749999998 podStartE2EDuration="1.750765375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:09:12.749277129 +0000 UTC m=+2029.903355416" watchObservedRunningTime="2026-09-16 15:09:12.750765375 +0000 UTC m=+2029.904843664"
Sep 16 15:09:13.735018 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:13.734980    2558 generic.go:356] "Generic (PLEG): container finished" podID="1814869e-f91b-4d8c-a56e-6328fc7bba88" containerID="ac27aa85902d2bf6fc91439b104c22f25acf1c517e67106af1ec084321ac3205" exitCode=0
Sep 16 15:09:13.735545 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:13.735069    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z" event={"ID":"1814869e-f91b-4d8c-a56e-6328fc7bba88","Type":"ContainerDied","Data":"ac27aa85902d2bf6fc91439b104c22f25acf1c517e67106af1ec084321ac3205"}
Sep 16 15:09:14.869444 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:14.869420    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"
Sep 16 15:09:14.943864 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:14.943830    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1814869e-f91b-4d8c-a56e-6328fc7bba88-maas-api-service-ca\") pod \"1814869e-f91b-4d8c-a56e-6328fc7bba88\" (UID: \"1814869e-f91b-4d8c-a56e-6328fc7bba88\") "
Sep 16 15:09:14.944216 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:14.944160    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1814869e-f91b-4d8c-a56e-6328fc7bba88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1814869e-f91b-4d8c-a56e-6328fc7bba88" (UID: "1814869e-f91b-4d8c-a56e-6328fc7bba88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:09:15.044813 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:15.044729    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1814869e-f91b-4d8c-a56e-6328fc7bba88-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:09:15.746946 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:15.746858    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"
Sep 16 15:09:15.746946 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:15.746906    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z" event={"ID":"1814869e-f91b-4d8c-a56e-6328fc7bba88","Type":"ContainerDied","Data":"4c2b4f0ae3688f61145147190000aed2cc012292649f8c14b72937c199ac9ec7"}
Sep 16 15:09:15.747129 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:15.746954    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c2b4f0ae3688f61145147190000aed2cc012292649f8c14b72937c199ac9ec7"
Sep 16 15:09:23.754482 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:23.754454    2558 scope.go:117] "RemoveContainer" containerID="a061e0034dd839a6d814c0c3b1e776b953d8452b825d05d292f900a40e8a5b19"
Sep 16 15:09:23.763441 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:23.763426    2558 scope.go:117] "RemoveContainer" containerID="7df868eb3d6e1cdb34feaf27294037de85d09e40534b724d3e9d82ceab0de253"
Sep 16 15:09:23.770862 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:23.770842    2558 scope.go:117] "RemoveContainer" containerID="c774adccffe428ee718690e10e8fa81322defc9f339b235334b21e5218faa8a2"
Sep 16 15:09:27.419577 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.419546    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"]
Sep 16 15:09:27.419946 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.419760    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd" podUID="1c8d8b9b-4471-46a1-ad34-1eed71fbe26d" containerName="payload-processing" containerID="cri-o://e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95" gracePeriod=30
Sep 16 15:09:27.684596 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.684522    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:09:27.744970 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.744944    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjfpw\" (UniqueName: \"kubernetes.io/projected/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-kube-api-access-bjfpw\") pod \"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d\" (UID: \"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d\") "
Sep 16 15:09:27.745138 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.745065    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-plugins-config-volume\") pod \"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d\" (UID: \"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d\") "
Sep 16 15:09:27.745509 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.745478    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c8d8b9b-4471-46a1-ad34-1eed71fbe26d" (UID: "1c8d8b9b-4471-46a1-ad34-1eed71fbe26d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:09:27.746984 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.746964    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-kube-api-access-bjfpw" (OuterVolumeSpecName: "kube-api-access-bjfpw") pod "1c8d8b9b-4471-46a1-ad34-1eed71fbe26d" (UID: "1c8d8b9b-4471-46a1-ad34-1eed71fbe26d"). InnerVolumeSpecName "kube-api-access-bjfpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:09:27.802611 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.802580    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"]
Sep 16 15:09:27.811780 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.811749    2558 generic.go:356] "Generic (PLEG): container finished" podID="1c8d8b9b-4471-46a1-ad34-1eed71fbe26d" containerID="e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95" exitCode=0
Sep 16 15:09:27.811916 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.811792    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"
Sep 16 15:09:27.811916 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.811790    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd" event={"ID":"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d","Type":"ContainerDied","Data":"e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95"}
Sep 16 15:09:27.812015 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.811915    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd" event={"ID":"1c8d8b9b-4471-46a1-ad34-1eed71fbe26d","Type":"ContainerDied","Data":"e9bd31b8b2fc7ffd787e1d3029b640e351af9bb5c4d564bc7a1c888b4edf0cba"}
Sep 16 15:09:27.812015 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.811944    2558 scope.go:117] "RemoveContainer" containerID="e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95"
Sep 16 15:09:27.812363 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.812338    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-91f4ab-lkc8z"]
Sep 16 15:09:27.820719 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.820705    2558 scope.go:117] "RemoveContainer" containerID="e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95"
Sep 16 15:09:27.820958 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:09:27.820943    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95\": container with ID starting with e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95 not found: ID does not exist" containerID="e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95"
Sep 16 15:09:27.820994 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.820966    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95"} err="failed to get container status \"e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95\": rpc error: code = NotFound desc = could not find container \"e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95\": container with ID starting with e88d264caefb7084a3e1db85f9a65f83548fd722f9c63d22c10c4857a14b5c95 not found: ID does not exist"
Sep 16 15:09:27.845776 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.845754    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:09:27.845776 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.845777    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfpw\" (UniqueName: \"kubernetes.io/projected/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d-kube-api-access-bjfpw\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:09:27.861173 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.861137    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"]
Sep 16 15:09:27.863098 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:27.863074    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-91f4ab-7d967d8ff9-5tltd"]
Sep 16 15:09:29.295826 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:29.295795    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1814869e-f91b-4d8c-a56e-6328fc7bba88" path="/var/lib/kubelet/pods/1814869e-f91b-4d8c-a56e-6328fc7bba88/volumes"
Sep 16 15:09:29.296227 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:09:29.296212    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c8d8b9b-4471-46a1-ad34-1eed71fbe26d" path="/var/lib/kubelet/pods/1c8d8b9b-4471-46a1-ad34-1eed71fbe26d/volumes"
Sep 16 15:10:18.158065 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.158031    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"]
Sep 16 15:10:18.158527 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.158449    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c8d8b9b-4471-46a1-ad34-1eed71fbe26d" containerName="payload-processing"
Sep 16 15:10:18.158527 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.158461    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c8d8b9b-4471-46a1-ad34-1eed71fbe26d" containerName="payload-processing"
Sep 16 15:10:18.158527 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.158471    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1814869e-f91b-4d8c-a56e-6328fc7bba88" containerName="revoke-keys"
Sep 16 15:10:18.158527 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.158476    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1814869e-f91b-4d8c-a56e-6328fc7bba88" containerName="revoke-keys"
Sep 16 15:10:18.158668 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.158532    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c8d8b9b-4471-46a1-ad34-1eed71fbe26d" containerName="payload-processing"
Sep 16 15:10:18.158668 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.158543    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1814869e-f91b-4d8c-a56e-6328fc7bba88" containerName="revoke-keys"
Sep 16 15:10:18.161485 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.161468    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"
Sep 16 15:10:18.164467 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.164451    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:10:18.175552 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.175533    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"]
Sep 16 15:10:18.216200 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.216155    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b29763-a60a-46ad-add7-37ff8586625b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv\" (UID: \"e2b29763-a60a-46ad-add7-37ff8586625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"
Sep 16 15:10:18.317008 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.316970    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b29763-a60a-46ad-add7-37ff8586625b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv\" (UID: \"e2b29763-a60a-46ad-add7-37ff8586625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"
Sep 16 15:10:18.317656 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.317635    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b29763-a60a-46ad-add7-37ff8586625b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv\" (UID: \"e2b29763-a60a-46ad-add7-37ff8586625b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"
Sep 16 15:10:18.471093 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.471005    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"
Sep 16 15:10:18.765975 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:18.765898    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"]
Sep 16 15:10:18.769751 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:10:18.769727    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b29763_a60a_46ad_add7_37ff8586625b.slice/crio-536c51d60c53b97fc92128e0501c8970ed5e20dc4ba336c151f6576dc5cc01d3 WatchSource:0}: Error finding container 536c51d60c53b97fc92128e0501c8970ed5e20dc4ba336c151f6576dc5cc01d3: Status 404 returned error can't find the container with id 536c51d60c53b97fc92128e0501c8970ed5e20dc4ba336c151f6576dc5cc01d3
Sep 16 15:10:19.072985 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:19.072958    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv" event={"ID":"e2b29763-a60a-46ad-add7-37ff8586625b","Type":"ContainerStarted","Data":"1ae963edf4d124ffc5703aa0d8e21afa630da932538899a46910ca2ce1814758"}
Sep 16 15:10:19.073105 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:19.073006    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv" event={"ID":"e2b29763-a60a-46ad-add7-37ff8586625b","Type":"ContainerStarted","Data":"536c51d60c53b97fc92128e0501c8970ed5e20dc4ba336c151f6576dc5cc01d3"}
Sep 16 15:10:19.090425 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:19.090376    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv" podStartSLOduration=1.090359573 podStartE2EDuration="1.090359573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:10:19.088071957 +0000 UTC m=+2096.242150241" watchObservedRunningTime="2026-09-16 15:10:19.090359573 +0000 UTC m=+2096.244437859"
Sep 16 15:10:20.080207 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:20.080160    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2b29763-a60a-46ad-add7-37ff8586625b" containerID="1ae963edf4d124ffc5703aa0d8e21afa630da932538899a46910ca2ce1814758" exitCode=0
Sep 16 15:10:20.080577 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:20.080252    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv" event={"ID":"e2b29763-a60a-46ad-add7-37ff8586625b","Type":"ContainerDied","Data":"1ae963edf4d124ffc5703aa0d8e21afa630da932538899a46910ca2ce1814758"}
Sep 16 15:10:21.214710 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:21.214688    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"
Sep 16 15:10:21.243792 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:21.243771    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b29763-a60a-46ad-add7-37ff8586625b-maas-api-service-ca\") pod \"e2b29763-a60a-46ad-add7-37ff8586625b\" (UID: \"e2b29763-a60a-46ad-add7-37ff8586625b\") "
Sep 16 15:10:21.244110 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:21.244090    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b29763-a60a-46ad-add7-37ff8586625b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2b29763-a60a-46ad-add7-37ff8586625b" (UID: "e2b29763-a60a-46ad-add7-37ff8586625b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:10:21.344877 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:21.344807    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b29763-a60a-46ad-add7-37ff8586625b-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:10:22.090903 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:22.090874    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"
Sep 16 15:10:22.091099 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:22.090907    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv" event={"ID":"e2b29763-a60a-46ad-add7-37ff8586625b","Type":"ContainerDied","Data":"536c51d60c53b97fc92128e0501c8970ed5e20dc4ba336c151f6576dc5cc01d3"}
Sep 16 15:10:22.091099 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:22.090949    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="536c51d60c53b97fc92128e0501c8970ed5e20dc4ba336c151f6576dc5cc01d3"
Sep 16 15:10:23.364091 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:23.364059    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 15:10:23.372199 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:23.372156    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 15:10:23.803115 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:23.803095    2558 scope.go:117] "RemoveContainer" containerID="df2d4e8ec34a4c68c92a72e2be3beb8d2dbeea35f21804764cfb02e1c19e4054"
Sep 16 15:10:23.811234 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:23.811210    2558 scope.go:117] "RemoveContainer" containerID="5054e31ae469e82c17a685aa3904580be17326f892146be85516a903c688caf5"
Sep 16 15:10:28.787131 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:28.787102    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"]
Sep 16 15:10:28.787647 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:28.787336    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6" podUID="36c3dffb-4097-4373-9119-99aa4a01577b" containerName="payload-processing" containerID="cri-o://e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de" gracePeriod=30
Sep 16 15:10:28.800563 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:28.800526    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"]
Sep 16 15:10:28.800791 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:28.800749    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl" podUID="cd801944-eefe-47e5-9ae1-9cb1c9b93a6a" containerName="payload-pre-processing" containerID="cri-o://9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6" gracePeriod=30
Sep 16 15:10:29.053802 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.053769    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"]
Sep 16 15:10:29.059299 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.059272    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-438a9b-2zxbv"]
Sep 16 15:10:29.079220 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.079196    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:10:29.085215 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.085195    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:10:29.127443 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.126969    2558 generic.go:356] "Generic (PLEG): container finished" podID="cd801944-eefe-47e5-9ae1-9cb1c9b93a6a" containerID="9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6" exitCode=0
Sep 16 15:10:29.127443 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.127235    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"
Sep 16 15:10:29.127443 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.127316    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl" event={"ID":"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a","Type":"ContainerDied","Data":"9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6"}
Sep 16 15:10:29.127761 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.127454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl" event={"ID":"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a","Type":"ContainerDied","Data":"79b252041d1d79b3470039a8196e50486792f653b12ba31a3e90b838fa90f847"}
Sep 16 15:10:29.127761 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.127486    2558 scope.go:117] "RemoveContainer" containerID="9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6"
Sep 16 15:10:29.130184 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.130138    2558 generic.go:356] "Generic (PLEG): container finished" podID="36c3dffb-4097-4373-9119-99aa4a01577b" containerID="e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de" exitCode=0
Sep 16 15:10:29.130312 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.130211    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"
Sep 16 15:10:29.130312 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.130212    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6" event={"ID":"36c3dffb-4097-4373-9119-99aa4a01577b","Type":"ContainerDied","Data":"e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de"}
Sep 16 15:10:29.130312 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.130261    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6" event={"ID":"36c3dffb-4097-4373-9119-99aa4a01577b","Type":"ContainerDied","Data":"800bf64f703de06202e258ebc6ea48d2a88dfd0ec38a0e27dd7053001620b51c"}
Sep 16 15:10:29.140049 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.140026    2558 scope.go:117] "RemoveContainer" containerID="9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6"
Sep 16 15:10:29.140472 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:10:29.140447    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6\": container with ID starting with 9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6 not found: ID does not exist" containerID="9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6"
Sep 16 15:10:29.140563 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.140490    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6"} err="failed to get container status \"9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6\": rpc error: code = NotFound desc = could not find container \"9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6\": container with ID starting with 9ffac99d0d6a2b7633d94168dd8417a2c9b7ab0c94fc1765bb11b41a6578a6c6 not found: ID does not exist"
Sep 16 15:10:29.140563 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.140522    2558 scope.go:117] "RemoveContainer" containerID="e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de"
Sep 16 15:10:29.149578 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.149555    2558 scope.go:117] "RemoveContainer" containerID="e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de"
Sep 16 15:10:29.149932 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:10:29.149913    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de\": container with ID starting with e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de not found: ID does not exist" containerID="e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de"
Sep 16 15:10:29.150031 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.149939    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de"} err="failed to get container status \"e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de\": rpc error: code = NotFound desc = could not find container \"e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de\": container with ID starting with e950284b2952b29361c558b72c7b7316324e51b4968c9097fd7ec89bd02a22de not found: ID does not exist"
Sep 16 15:10:29.220788 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.220756    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4rdp\" (UniqueName: \"kubernetes.io/projected/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-kube-api-access-n4rdp\") pod \"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a\" (UID: \"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a\") "
Sep 16 15:10:29.220958 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.220807    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c3dffb-4097-4373-9119-99aa4a01577b-plugins-config-volume\") pod \"36c3dffb-4097-4373-9119-99aa4a01577b\" (UID: \"36c3dffb-4097-4373-9119-99aa4a01577b\") "
Sep 16 15:10:29.220958 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.220838    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-plugins-config-volume\") pod \"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a\" (UID: \"cd801944-eefe-47e5-9ae1-9cb1c9b93a6a\") "
Sep 16 15:10:29.221045 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.221001    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvzfq\" (UniqueName: \"kubernetes.io/projected/36c3dffb-4097-4373-9119-99aa4a01577b-kube-api-access-bvzfq\") pod \"36c3dffb-4097-4373-9119-99aa4a01577b\" (UID: \"36c3dffb-4097-4373-9119-99aa4a01577b\") "
Sep 16 15:10:29.221271 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.221242    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd801944-eefe-47e5-9ae1-9cb1c9b93a6a" (UID: "cd801944-eefe-47e5-9ae1-9cb1c9b93a6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:10:29.221271 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.221258    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c3dffb-4097-4373-9119-99aa4a01577b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36c3dffb-4097-4373-9119-99aa4a01577b" (UID: "36c3dffb-4097-4373-9119-99aa4a01577b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:10:29.221409 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.221368    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36c3dffb-4097-4373-9119-99aa4a01577b-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:10:29.221409 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.221383    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:10:29.223066 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.223038    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c3dffb-4097-4373-9119-99aa4a01577b-kube-api-access-bvzfq" (OuterVolumeSpecName: "kube-api-access-bvzfq") pod "36c3dffb-4097-4373-9119-99aa4a01577b" (UID: "36c3dffb-4097-4373-9119-99aa4a01577b"). InnerVolumeSpecName "kube-api-access-bvzfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:10:29.223066 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.223056    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-kube-api-access-n4rdp" (OuterVolumeSpecName: "kube-api-access-n4rdp") pod "cd801944-eefe-47e5-9ae1-9cb1c9b93a6a" (UID: "cd801944-eefe-47e5-9ae1-9cb1c9b93a6a"). InnerVolumeSpecName "kube-api-access-n4rdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:10:29.295182 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.295135    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2b29763-a60a-46ad-add7-37ff8586625b" path="/var/lib/kubelet/pods/e2b29763-a60a-46ad-add7-37ff8586625b/volumes"
Sep 16 15:10:29.322640 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.322605    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4rdp\" (UniqueName: \"kubernetes.io/projected/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a-kube-api-access-n4rdp\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:10:29.322640 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.322633    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvzfq\" (UniqueName: \"kubernetes.io/projected/36c3dffb-4097-4373-9119-99aa4a01577b-kube-api-access-bvzfq\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:10:29.447071 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.447030    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"]
Sep 16 15:10:29.453466 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.453438    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-438a9b-84f5db845d-bmrnl"]
Sep 16 15:10:29.465908 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.465878    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"]
Sep 16 15:10:29.478177 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:29.478135    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-438a9b-6946c5b445-f42j6"]
Sep 16 15:10:31.295072 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:31.295039    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36c3dffb-4097-4373-9119-99aa4a01577b" path="/var/lib/kubelet/pods/36c3dffb-4097-4373-9119-99aa4a01577b/volumes"
Sep 16 15:10:31.295472 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:31.295458    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd801944-eefe-47e5-9ae1-9cb1c9b93a6a" path="/var/lib/kubelet/pods/cd801944-eefe-47e5-9ae1-9cb1c9b93a6a/volumes"
Sep 16 15:10:37.424012 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.423979    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"]
Sep 16 15:10:37.424414 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.424340    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd801944-eefe-47e5-9ae1-9cb1c9b93a6a" containerName="payload-pre-processing"
Sep 16 15:10:37.424414 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.424352    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd801944-eefe-47e5-9ae1-9cb1c9b93a6a" containerName="payload-pre-processing"
Sep 16 15:10:37.424414 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.424373    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b29763-a60a-46ad-add7-37ff8586625b" containerName="revoke-keys"
Sep 16 15:10:37.424414 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.424379    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b29763-a60a-46ad-add7-37ff8586625b" containerName="revoke-keys"
Sep 16 15:10:37.424414 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.424387    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36c3dffb-4097-4373-9119-99aa4a01577b" containerName="payload-processing"
Sep 16 15:10:37.424414 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.424393    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="36c3dffb-4097-4373-9119-99aa4a01577b" containerName="payload-processing"
Sep 16 15:10:37.424615 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.424445    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="36c3dffb-4097-4373-9119-99aa4a01577b" containerName="payload-processing"
Sep 16 15:10:37.424615 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.424453    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd801944-eefe-47e5-9ae1-9cb1c9b93a6a" containerName="payload-pre-processing"
Sep 16 15:10:37.424615 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.424463    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2b29763-a60a-46ad-add7-37ff8586625b" containerName="revoke-keys"
Sep 16 15:10:37.428730 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.428709    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"
Sep 16 15:10:37.431519 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.431495    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:10:37.440098 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.440062    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"]
Sep 16 15:10:37.506241 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.506204    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc3782a-ed22-4882-8ed3-2c1bd0118683-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2\" (UID: \"ddc3782a-ed22-4882-8ed3-2c1bd0118683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"
Sep 16 15:10:37.606697 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.606660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc3782a-ed22-4882-8ed3-2c1bd0118683-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2\" (UID: \"ddc3782a-ed22-4882-8ed3-2c1bd0118683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"
Sep 16 15:10:37.607305 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.607286    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc3782a-ed22-4882-8ed3-2c1bd0118683-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2\" (UID: \"ddc3782a-ed22-4882-8ed3-2c1bd0118683\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"
Sep 16 15:10:37.739544 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:37.739471    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"
Sep 16 15:10:38.021552 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:38.021528    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"]
Sep 16 15:10:38.024032 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:10:38.024004    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc3782a_ed22_4882_8ed3_2c1bd0118683.slice/crio-1c9ba7b9c25018b9935722c698e7912978c5e0d9c2d829891f9377ae0b734620 WatchSource:0}: Error finding container 1c9ba7b9c25018b9935722c698e7912978c5e0d9c2d829891f9377ae0b734620: Status 404 returned error can't find the container with id 1c9ba7b9c25018b9935722c698e7912978c5e0d9c2d829891f9377ae0b734620
Sep 16 15:10:38.176762 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:38.176731    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2" event={"ID":"ddc3782a-ed22-4882-8ed3-2c1bd0118683","Type":"ContainerStarted","Data":"1c9ba7b9c25018b9935722c698e7912978c5e0d9c2d829891f9377ae0b734620"}
Sep 16 15:10:39.183103 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:39.183069    2558 generic.go:356] "Generic (PLEG): container finished" podID="ddc3782a-ed22-4882-8ed3-2c1bd0118683" containerID="c1d3d6421a2bf47a55ff9793b1dc408de1fe3257746b633812928ff9b7b2572d" exitCode=0
Sep 16 15:10:39.183501 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:39.183114    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2" event={"ID":"ddc3782a-ed22-4882-8ed3-2c1bd0118683","Type":"ContainerDied","Data":"c1d3d6421a2bf47a55ff9793b1dc408de1fe3257746b633812928ff9b7b2572d"}
Sep 16 15:10:40.316901 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:40.316876    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"
Sep 16 15:10:40.433033 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:40.432996    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc3782a-ed22-4882-8ed3-2c1bd0118683-maas-api-service-ca\") pod \"ddc3782a-ed22-4882-8ed3-2c1bd0118683\" (UID: \"ddc3782a-ed22-4882-8ed3-2c1bd0118683\") "
Sep 16 15:10:40.433405 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:40.433381    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc3782a-ed22-4882-8ed3-2c1bd0118683-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddc3782a-ed22-4882-8ed3-2c1bd0118683" (UID: "ddc3782a-ed22-4882-8ed3-2c1bd0118683"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:10:40.534613 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:40.534522    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc3782a-ed22-4882-8ed3-2c1bd0118683-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:10:41.193704 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:41.193678    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"
Sep 16 15:10:41.193879 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:41.193710    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2" event={"ID":"ddc3782a-ed22-4882-8ed3-2c1bd0118683","Type":"ContainerDied","Data":"1c9ba7b9c25018b9935722c698e7912978c5e0d9c2d829891f9377ae0b734620"}
Sep 16 15:10:41.193879 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:41.193755    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c9ba7b9c25018b9935722c698e7912978c5e0d9c2d829891f9377ae0b734620"
Sep 16 15:10:48.051129 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.051092    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm"]
Sep 16 15:10:48.055716 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.055694    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d28bd5-29826180-wwbtm"]
Sep 16 15:10:48.091740 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.091711    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"]
Sep 16 15:10:48.091972 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.091917    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl" podUID="fb16db93-b470-4026-ad63-f4210fa03565" containerName="payload-pre-processing" containerID="cri-o://673e9d63bf88afcbdf469c1182071b9e584ec2fa5c33639e37ee234d0c64ba97" gracePeriod=30
Sep 16 15:10:48.231108 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.231070    2558 generic.go:356] "Generic (PLEG): container finished" podID="fb16db93-b470-4026-ad63-f4210fa03565" containerID="673e9d63bf88afcbdf469c1182071b9e584ec2fa5c33639e37ee234d0c64ba97" exitCode=0
Sep 16 15:10:48.231309 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.231219    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl" event={"ID":"fb16db93-b470-4026-ad63-f4210fa03565","Type":"ContainerDied","Data":"673e9d63bf88afcbdf469c1182071b9e584ec2fa5c33639e37ee234d0c64ba97"}
Sep 16 15:10:48.323281 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.323247    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"]
Sep 16 15:10:48.329563 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.329542    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d28bd5-cdsw2"]
Sep 16 15:10:48.349824 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.349801    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 15:10:48.405204 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.405160    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb16db93-b470-4026-ad63-f4210fa03565-plugins-config-volume\") pod \"fb16db93-b470-4026-ad63-f4210fa03565\" (UID: \"fb16db93-b470-4026-ad63-f4210fa03565\") "
Sep 16 15:10:48.405362 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.405252    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqqb7\" (UniqueName: \"kubernetes.io/projected/fb16db93-b470-4026-ad63-f4210fa03565-kube-api-access-vqqb7\") pod \"fb16db93-b470-4026-ad63-f4210fa03565\" (UID: \"fb16db93-b470-4026-ad63-f4210fa03565\") "
Sep 16 15:10:48.405478 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.405456    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb16db93-b470-4026-ad63-f4210fa03565-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb16db93-b470-4026-ad63-f4210fa03565" (UID: "fb16db93-b470-4026-ad63-f4210fa03565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:10:48.407176 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.407138    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb16db93-b470-4026-ad63-f4210fa03565-kube-api-access-vqqb7" (OuterVolumeSpecName: "kube-api-access-vqqb7") pod "fb16db93-b470-4026-ad63-f4210fa03565" (UID: "fb16db93-b470-4026-ad63-f4210fa03565"). InnerVolumeSpecName "kube-api-access-vqqb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:10:48.506262 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.506225    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb16db93-b470-4026-ad63-f4210fa03565-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:10:48.506262 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:48.506253    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqqb7\" (UniqueName: \"kubernetes.io/projected/fb16db93-b470-4026-ad63-f4210fa03565-kube-api-access-vqqb7\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:10:49.238433 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:49.238403    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"
Sep 16 15:10:49.238433 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:49.238430    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl" event={"ID":"fb16db93-b470-4026-ad63-f4210fa03565","Type":"ContainerDied","Data":"37d75814498e8196f22bc34225d7734a089d1426d9570eb408a4ffc8c4b7abfb"}
Sep 16 15:10:49.239011 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:49.238494    2558 scope.go:117] "RemoveContainer" containerID="673e9d63bf88afcbdf469c1182071b9e584ec2fa5c33639e37ee234d0c64ba97"
Sep 16 15:10:49.265121 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:49.265093    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"]
Sep 16 15:10:49.269486 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:49.269461    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d28bd5-549777cbb-tdxgl"]
Sep 16 15:10:49.295416 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:49.295390    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13508f81-3522-4230-9177-a6c78ff2f773" path="/var/lib/kubelet/pods/13508f81-3522-4230-9177-a6c78ff2f773/volumes"
Sep 16 15:10:49.295701 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:49.295689    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddc3782a-ed22-4882-8ed3-2c1bd0118683" path="/var/lib/kubelet/pods/ddc3782a-ed22-4882-8ed3-2c1bd0118683/volumes"
Sep 16 15:10:49.296003 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:49.295992    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb16db93-b470-4026-ad63-f4210fa03565" path="/var/lib/kubelet/pods/fb16db93-b470-4026-ad63-f4210fa03565/volumes"
Sep 16 15:10:55.718394 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.718343    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"]
Sep 16 15:10:55.718846 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.718761    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc3782a-ed22-4882-8ed3-2c1bd0118683" containerName="revoke-keys"
Sep 16 15:10:55.718846 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.718779    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc3782a-ed22-4882-8ed3-2c1bd0118683" containerName="revoke-keys"
Sep 16 15:10:55.718846 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.718826    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb16db93-b470-4026-ad63-f4210fa03565" containerName="payload-pre-processing"
Sep 16 15:10:55.718846 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.718836    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb16db93-b470-4026-ad63-f4210fa03565" containerName="payload-pre-processing"
Sep 16 15:10:55.719017 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.718926    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb16db93-b470-4026-ad63-f4210fa03565" containerName="payload-pre-processing"
Sep 16 15:10:55.719017 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.718938    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc3782a-ed22-4882-8ed3-2c1bd0118683" containerName="revoke-keys"
Sep 16 15:10:55.723351 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.723335    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"
Sep 16 15:10:55.726062 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.726045    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:10:55.733597 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.733577    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"]
Sep 16 15:10:55.763080 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.763041    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5a923e-0974-4580-ae22-1e84974af42c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92\" (UID: \"ae5a923e-0974-4580-ae22-1e84974af42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"
Sep 16 15:10:55.863995 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.863948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5a923e-0974-4580-ae22-1e84974af42c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92\" (UID: \"ae5a923e-0974-4580-ae22-1e84974af42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"
Sep 16 15:10:55.864640 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:55.864613    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5a923e-0974-4580-ae22-1e84974af42c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92\" (UID: \"ae5a923e-0974-4580-ae22-1e84974af42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"
Sep 16 15:10:56.034820 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:56.034722    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"
Sep 16 15:10:56.308791 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:56.308767    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"]
Sep 16 15:10:56.311203 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:10:56.311152    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5a923e_0974_4580_ae22_1e84974af42c.slice/crio-9d0bc226b641d944d609fc81757b2bef2401bb91648d482fba184b11f3ace6e0 WatchSource:0}: Error finding container 9d0bc226b641d944d609fc81757b2bef2401bb91648d482fba184b11f3ace6e0: Status 404 returned error can't find the container with id 9d0bc226b641d944d609fc81757b2bef2401bb91648d482fba184b11f3ace6e0
Sep 16 15:10:57.278911 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:57.278817    2558 generic.go:356] "Generic (PLEG): container finished" podID="ae5a923e-0974-4580-ae22-1e84974af42c" containerID="0260f844c3cf1a5bb8cc8fcd7a8d259f787cdc8398b3a17edc0da883aa0b15eb" exitCode=0
Sep 16 15:10:57.278911 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:57.278871    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92" event={"ID":"ae5a923e-0974-4580-ae22-1e84974af42c","Type":"ContainerDied","Data":"0260f844c3cf1a5bb8cc8fcd7a8d259f787cdc8398b3a17edc0da883aa0b15eb"}
Sep 16 15:10:57.279322 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:57.278915    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92" event={"ID":"ae5a923e-0974-4580-ae22-1e84974af42c","Type":"ContainerStarted","Data":"9d0bc226b641d944d609fc81757b2bef2401bb91648d482fba184b11f3ace6e0"}
Sep 16 15:10:58.413359 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:58.413328    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"
Sep 16 15:10:58.491430 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:58.491400    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5a923e-0974-4580-ae22-1e84974af42c-maas-api-service-ca\") pod \"ae5a923e-0974-4580-ae22-1e84974af42c\" (UID: \"ae5a923e-0974-4580-ae22-1e84974af42c\") "
Sep 16 15:10:58.491759 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:58.491735    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae5a923e-0974-4580-ae22-1e84974af42c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae5a923e-0974-4580-ae22-1e84974af42c" (UID: "ae5a923e-0974-4580-ae22-1e84974af42c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:10:58.592420 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:58.592388    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae5a923e-0974-4580-ae22-1e84974af42c-maas-api-service-ca\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:10:59.290894 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:59.290864    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"
Sep 16 15:10:59.295695 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:59.295665    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92" event={"ID":"ae5a923e-0974-4580-ae22-1e84974af42c","Type":"ContainerDied","Data":"9d0bc226b641d944d609fc81757b2bef2401bb91648d482fba184b11f3ace6e0"}
Sep 16 15:10:59.295823 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:10:59.295711    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d0bc226b641d944d609fc81757b2bef2401bb91648d482fba184b11f3ace6e0"
Sep 16 15:11:06.304722 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.304685    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"]
Sep 16 15:11:06.307797 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.304951    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs" podUID="af85bb43-ea5c-4233-bee0-d597a4d6b8d3" containerName="payload-processing" containerID="cri-o://3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6" gracePeriod=30
Sep 16 15:11:06.310489 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.310463    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"]
Sep 16 15:11:06.311343 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.311275    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp" podUID="19309040-11c1-4f60-870b-b60ea17febd5" containerName="payload-pre-processing" containerID="cri-o://1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759" gracePeriod=30
Sep 16 15:11:06.498827 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.498793    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"]
Sep 16 15:11:06.513396 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.513368    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-cadcf4-zpx92"]
Sep 16 15:11:06.577648 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.577622    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 15:11:06.581400 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.581375    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 15:11:06.661916 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.661886    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19309040-11c1-4f60-870b-b60ea17febd5-plugins-config-volume\") pod \"19309040-11c1-4f60-870b-b60ea17febd5\" (UID: \"19309040-11c1-4f60-870b-b60ea17febd5\") "
Sep 16 15:11:06.662132 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.661960    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-plugins-config-volume\") pod \"af85bb43-ea5c-4233-bee0-d597a4d6b8d3\" (UID: \"af85bb43-ea5c-4233-bee0-d597a4d6b8d3\") "
Sep 16 15:11:06.662132 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.662004    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkwtt\" (UniqueName: \"kubernetes.io/projected/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-kube-api-access-pkwtt\") pod \"af85bb43-ea5c-4233-bee0-d597a4d6b8d3\" (UID: \"af85bb43-ea5c-4233-bee0-d597a4d6b8d3\") "
Sep 16 15:11:06.662132 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.662046    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g224q\" (UniqueName: \"kubernetes.io/projected/19309040-11c1-4f60-870b-b60ea17febd5-kube-api-access-g224q\") pod \"19309040-11c1-4f60-870b-b60ea17febd5\" (UID: \"19309040-11c1-4f60-870b-b60ea17febd5\") "
Sep 16 15:11:06.662405 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.662362    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19309040-11c1-4f60-870b-b60ea17febd5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19309040-11c1-4f60-870b-b60ea17febd5" (UID: "19309040-11c1-4f60-870b-b60ea17febd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:11:06.662405 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.662363    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af85bb43-ea5c-4233-bee0-d597a4d6b8d3" (UID: "af85bb43-ea5c-4233-bee0-d597a4d6b8d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 15:11:06.664225 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.664187    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-kube-api-access-pkwtt" (OuterVolumeSpecName: "kube-api-access-pkwtt") pod "af85bb43-ea5c-4233-bee0-d597a4d6b8d3" (UID: "af85bb43-ea5c-4233-bee0-d597a4d6b8d3"). InnerVolumeSpecName "kube-api-access-pkwtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:11:06.664521 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.664494    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19309040-11c1-4f60-870b-b60ea17febd5-kube-api-access-g224q" (OuterVolumeSpecName: "kube-api-access-g224q") pod "19309040-11c1-4f60-870b-b60ea17febd5" (UID: "19309040-11c1-4f60-870b-b60ea17febd5"). InnerVolumeSpecName "kube-api-access-g224q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:11:06.763233 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.763198    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19309040-11c1-4f60-870b-b60ea17febd5-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:11:06.763233 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.763232    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-plugins-config-volume\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:11:06.763443 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.763245    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkwtt\" (UniqueName: \"kubernetes.io/projected/af85bb43-ea5c-4233-bee0-d597a4d6b8d3-kube-api-access-pkwtt\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:11:06.763443 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:06.763258    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g224q\" (UniqueName: \"kubernetes.io/projected/19309040-11c1-4f60-870b-b60ea17febd5-kube-api-access-g224q\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:11:07.294593 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.294556    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae5a923e-0974-4580-ae22-1e84974af42c" path="/var/lib/kubelet/pods/ae5a923e-0974-4580-ae22-1e84974af42c/volumes"
Sep 16 15:11:07.338277 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.338248    2558 generic.go:356] "Generic (PLEG): container finished" podID="19309040-11c1-4f60-870b-b60ea17febd5" containerID="1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759" exitCode=0
Sep 16 15:11:07.338634 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.338294    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"
Sep 16 15:11:07.338634 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.338325    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp" event={"ID":"19309040-11c1-4f60-870b-b60ea17febd5","Type":"ContainerDied","Data":"1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759"}
Sep 16 15:11:07.338634 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.338374    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp" event={"ID":"19309040-11c1-4f60-870b-b60ea17febd5","Type":"ContainerDied","Data":"b6466e47038ad5abec8b49522785194f20fe4a181539931bf3ec1edf7f946a41"}
Sep 16 15:11:07.338634 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.338394    2558 scope.go:117] "RemoveContainer" containerID="1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759"
Sep 16 15:11:07.339809 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.339784    2558 generic.go:356] "Generic (PLEG): container finished" podID="af85bb43-ea5c-4233-bee0-d597a4d6b8d3" containerID="3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6" exitCode=0
Sep 16 15:11:07.340310 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.339828    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs" event={"ID":"af85bb43-ea5c-4233-bee0-d597a4d6b8d3","Type":"ContainerDied","Data":"3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6"}
Sep 16 15:11:07.340310 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.339856    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"
Sep 16 15:11:07.340310 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.339867    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs" event={"ID":"af85bb43-ea5c-4233-bee0-d597a4d6b8d3","Type":"ContainerDied","Data":"6461d002938cc1d903744b8da42a9170c60cbbde108d5d0b9973278d0ee722db"}
Sep 16 15:11:07.347041 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.347018    2558 scope.go:117] "RemoveContainer" containerID="1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759"
Sep 16 15:11:07.347318 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:11:07.347300    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759\": container with ID starting with 1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759 not found: ID does not exist" containerID="1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759"
Sep 16 15:11:07.347408 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.347330    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759"} err="failed to get container status \"1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759\": rpc error: code = NotFound desc = could not find container \"1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759\": container with ID starting with 1befdcd9f24cb937065abc584e1da979987b2d4aec740112db279e066d168759 not found: ID does not exist"
Sep 16 15:11:07.347408 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.347352    2558 scope.go:117] "RemoveContainer" containerID="3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6"
Sep 16 15:11:07.354836 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.354820    2558 scope.go:117] "RemoveContainer" containerID="3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6"
Sep 16 15:11:07.355100 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:11:07.355081    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6\": container with ID starting with 3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6 not found: ID does not exist" containerID="3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6"
Sep 16 15:11:07.355275 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.355102    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6"} err="failed to get container status \"3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6\": rpc error: code = NotFound desc = could not find container \"3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6\": container with ID starting with 3b376df932761b471fc6f1a8a395fd99bee09b387277ee0b15c3f3a335bc49f6 not found: ID does not exist"
Sep 16 15:11:07.359840 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.359821    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"]
Sep 16 15:11:07.366137 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.366120    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-cadcf4-c7c96b846-j2whp"]
Sep 16 15:11:07.378088 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.378068    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"]
Sep 16 15:11:07.386715 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:07.386696    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cadcf4-9cb96d5bc-cr9gs"]
Sep 16 15:11:09.294700 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:09.294662    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19309040-11c1-4f60-870b-b60ea17febd5" path="/var/lib/kubelet/pods/19309040-11c1-4f60-870b-b60ea17febd5/volumes"
Sep 16 15:11:09.295153 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:09.295137    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af85bb43-ea5c-4233-bee0-d597a4d6b8d3" path="/var/lib/kubelet/pods/af85bb43-ea5c-4233-bee0-d597a4d6b8d3/volumes"
Sep 16 15:11:16.172835 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.172752    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-pdbbb"]
Sep 16 15:11:16.173360 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.172984    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb" podUID="15cab74b-508c-4f12-9e28-20e6adca900f" containerName="manager" containerID="cri-o://8f75986a3e7b2f6f4ff97322f6afde180377ccdddbdf62d22fd753bfdb64ec66" gracePeriod=30
Sep 16 15:11:16.387021 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.386980    2558 generic.go:356] "Generic (PLEG): container finished" podID="15cab74b-508c-4f12-9e28-20e6adca900f" containerID="8f75986a3e7b2f6f4ff97322f6afde180377ccdddbdf62d22fd753bfdb64ec66" exitCode=0
Sep 16 15:11:16.387221 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.387033    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb" event={"ID":"15cab74b-508c-4f12-9e28-20e6adca900f","Type":"ContainerDied","Data":"8f75986a3e7b2f6f4ff97322f6afde180377ccdddbdf62d22fd753bfdb64ec66"}
Sep 16 15:11:16.420426 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.420401    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 15:11:16.549121 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.549031    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcl98\" (UniqueName: \"kubernetes.io/projected/15cab74b-508c-4f12-9e28-20e6adca900f-kube-api-access-hcl98\") pod \"15cab74b-508c-4f12-9e28-20e6adca900f\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") "
Sep 16 15:11:16.549121 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.549086    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-metrics-certs\") pod \"15cab74b-508c-4f12-9e28-20e6adca900f\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") "
Sep 16 15:11:16.549121 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.549106    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-webhook-cert\") pod \"15cab74b-508c-4f12-9e28-20e6adca900f\" (UID: \"15cab74b-508c-4f12-9e28-20e6adca900f\") "
Sep 16 15:11:16.551226 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.551140    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "15cab74b-508c-4f12-9e28-20e6adca900f" (UID: "15cab74b-508c-4f12-9e28-20e6adca900f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:11:16.551226 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.551161    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "15cab74b-508c-4f12-9e28-20e6adca900f" (UID: "15cab74b-508c-4f12-9e28-20e6adca900f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:11:16.551376 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.551238    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cab74b-508c-4f12-9e28-20e6adca900f-kube-api-access-hcl98" (OuterVolumeSpecName: "kube-api-access-hcl98") pod "15cab74b-508c-4f12-9e28-20e6adca900f" (UID: "15cab74b-508c-4f12-9e28-20e6adca900f"). InnerVolumeSpecName "kube-api-access-hcl98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:11:16.650339 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.650302    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcl98\" (UniqueName: \"kubernetes.io/projected/15cab74b-508c-4f12-9e28-20e6adca900f-kube-api-access-hcl98\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:11:16.650339 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.650331    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-metrics-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:11:16.650339 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:16.650341    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15cab74b-508c-4f12-9e28-20e6adca900f-webhook-cert\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:11:17.394876 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:17.394840    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb"
Sep 16 15:11:17.395338 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:17.394839    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb4b58ccc-pdbbb" event={"ID":"15cab74b-508c-4f12-9e28-20e6adca900f","Type":"ContainerDied","Data":"bb02096cfeb5432cf67e4991824438f1e916e732a8d7ff6b5e4f866c9c5c6e28"}
Sep 16 15:11:17.395338 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:17.394962    2558 scope.go:117] "RemoveContainer" containerID="8f75986a3e7b2f6f4ff97322f6afde180377ccdddbdf62d22fd753bfdb64ec66"
Sep 16 15:11:17.413118 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:17.413091    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-pdbbb"]
Sep 16 15:11:17.417094 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:17.417070    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-pdbbb"]
Sep 16 15:11:19.294812 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:19.294779    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15cab74b-508c-4f12-9e28-20e6adca900f" path="/var/lib/kubelet/pods/15cab74b-508c-4f12-9e28-20e6adca900f/volumes"
Sep 16 15:11:23.839148 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:11:23.839105    2558 scope.go:117] "RemoveContainer" containerID="33ed67c29d6860eb3538d6142d186a29affee2dcefecce98824fd2203ab3b415"
Sep 16 15:12:18.249097 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249064    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-z9886"]
Sep 16 15:12:18.249674 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249573    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19309040-11c1-4f60-870b-b60ea17febd5" containerName="payload-pre-processing"
Sep 16 15:12:18.249674 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249593    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="19309040-11c1-4f60-870b-b60ea17febd5" containerName="payload-pre-processing"
Sep 16 15:12:18.249674 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249618    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af85bb43-ea5c-4233-bee0-d597a4d6b8d3" containerName="payload-processing"
Sep 16 15:12:18.249674 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249627    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="af85bb43-ea5c-4233-bee0-d597a4d6b8d3" containerName="payload-processing"
Sep 16 15:12:18.249674 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249639    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15cab74b-508c-4f12-9e28-20e6adca900f" containerName="manager"
Sep 16 15:12:18.249674 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249648    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="15cab74b-508c-4f12-9e28-20e6adca900f" containerName="manager"
Sep 16 15:12:18.249674 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249666    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae5a923e-0974-4580-ae22-1e84974af42c" containerName="revoke-keys"
Sep 16 15:12:18.249674 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249674    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae5a923e-0974-4580-ae22-1e84974af42c" containerName="revoke-keys"
Sep 16 15:12:18.250091 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249759    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="15cab74b-508c-4f12-9e28-20e6adca900f" containerName="manager"
Sep 16 15:12:18.250091 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249770    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="19309040-11c1-4f60-870b-b60ea17febd5" containerName="payload-pre-processing"
Sep 16 15:12:18.250091 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249783    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae5a923e-0974-4580-ae22-1e84974af42c" containerName="revoke-keys"
Sep 16 15:12:18.250091 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.249795    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="af85bb43-ea5c-4233-bee0-d597a4d6b8d3" containerName="payload-processing"
Sep 16 15:12:18.254301 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.254277    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:18.258538 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.258512    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 15:12:18.258751 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.258732    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tcknf\""
Sep 16 15:12:18.258984 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.258926    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 15:12:18.259245 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.259229    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 15:12:18.262752 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.262725    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-z9886"]
Sep 16 15:12:18.386592 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.386556    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-metrics-certs\") pod \"maas-controller-5cb4b58ccc-z9886\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") " pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:18.386592 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.386602    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-webhook-cert\") pod \"maas-controller-5cb4b58ccc-z9886\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") " pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:18.386841 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.386675    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9kcb\" (UniqueName: \"kubernetes.io/projected/49d268f1-8000-4609-8b3e-18570678c03b-kube-api-access-s9kcb\") pod \"maas-controller-5cb4b58ccc-z9886\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") " pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:18.487843 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.487811    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-metrics-certs\") pod \"maas-controller-5cb4b58ccc-z9886\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") " pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:18.488018 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.487850    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-webhook-cert\") pod \"maas-controller-5cb4b58ccc-z9886\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") " pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:18.488018 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.487875    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9kcb\" (UniqueName: \"kubernetes.io/projected/49d268f1-8000-4609-8b3e-18570678c03b-kube-api-access-s9kcb\") pod \"maas-controller-5cb4b58ccc-z9886\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") " pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:18.490264 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.490237    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-metrics-certs\") pod \"maas-controller-5cb4b58ccc-z9886\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") " pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:18.490375 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.490281    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-webhook-cert\") pod \"maas-controller-5cb4b58ccc-z9886\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") " pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:18.498271 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.498246    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9kcb\" (UniqueName: \"kubernetes.io/projected/49d268f1-8000-4609-8b3e-18570678c03b-kube-api-access-s9kcb\") pod \"maas-controller-5cb4b58ccc-z9886\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") " pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:18.568379 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.568354    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:18.842457 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.842435    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-z9886"]
Sep 16 15:12:18.844946 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:12:18.844922    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d268f1_8000_4609_8b3e_18570678c03b.slice/crio-1054dd95519961dd0e40ce6cb435ce659848ea5de3600ae96c49c6c8e63cc0dd WatchSource:0}: Error finding container 1054dd95519961dd0e40ce6cb435ce659848ea5de3600ae96c49c6c8e63cc0dd: Status 404 returned error can't find the container with id 1054dd95519961dd0e40ce6cb435ce659848ea5de3600ae96c49c6c8e63cc0dd
Sep 16 15:12:18.846848 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:18.846829    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 15:12:19.689602 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:19.689508    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb4b58ccc-z9886" event={"ID":"49d268f1-8000-4609-8b3e-18570678c03b","Type":"ContainerStarted","Data":"b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2"}
Sep 16 15:12:19.689602 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:19.689570    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb4b58ccc-z9886" event={"ID":"49d268f1-8000-4609-8b3e-18570678c03b","Type":"ContainerStarted","Data":"1054dd95519961dd0e40ce6cb435ce659848ea5de3600ae96c49c6c8e63cc0dd"}
Sep 16 15:12:19.690029 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:19.689636    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:19.709323 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:19.709268    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cb4b58ccc-z9886" podStartSLOduration=1.154142305 podStartE2EDuration="1.709247152s" totalImagesPullingTime="555.104847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:12:19.70802144 +0000 UTC m=+2216.862099728" watchObservedRunningTime="2026-09-16 15:12:19.709247152 +0000 UTC m=+2216.863325441"
Sep 16 15:12:30.697205 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:30.697155    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:40.001097 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.001061    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-z9886"]
Sep 16 15:12:40.001556 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.001350    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5cb4b58ccc-z9886" podUID="49d268f1-8000-4609-8b3e-18570678c03b" containerName="manager" containerID="cri-o://b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2" gracePeriod=30
Sep 16 15:12:40.240701 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.240677    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:40.279418 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.279327    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-webhook-cert\") pod \"49d268f1-8000-4609-8b3e-18570678c03b\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") "
Sep 16 15:12:40.279570 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.279460    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9kcb\" (UniqueName: \"kubernetes.io/projected/49d268f1-8000-4609-8b3e-18570678c03b-kube-api-access-s9kcb\") pod \"49d268f1-8000-4609-8b3e-18570678c03b\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") "
Sep 16 15:12:40.279570 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.279501    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-metrics-certs\") pod \"49d268f1-8000-4609-8b3e-18570678c03b\" (UID: \"49d268f1-8000-4609-8b3e-18570678c03b\") "
Sep 16 15:12:40.281550 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.281520    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d268f1-8000-4609-8b3e-18570678c03b-kube-api-access-s9kcb" (OuterVolumeSpecName: "kube-api-access-s9kcb") pod "49d268f1-8000-4609-8b3e-18570678c03b" (UID: "49d268f1-8000-4609-8b3e-18570678c03b"). InnerVolumeSpecName "kube-api-access-s9kcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:12:40.281658 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.281634    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "49d268f1-8000-4609-8b3e-18570678c03b" (UID: "49d268f1-8000-4609-8b3e-18570678c03b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:12:40.281870 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.281838    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "49d268f1-8000-4609-8b3e-18570678c03b" (UID: "49d268f1-8000-4609-8b3e-18570678c03b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:12:40.380536 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.380504    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9kcb\" (UniqueName: \"kubernetes.io/projected/49d268f1-8000-4609-8b3e-18570678c03b-kube-api-access-s9kcb\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:12:40.380536 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.380531    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-metrics-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:12:40.380536 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.380540    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49d268f1-8000-4609-8b3e-18570678c03b-webhook-cert\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:12:40.794404 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.794371    2558 generic.go:356] "Generic (PLEG): container finished" podID="49d268f1-8000-4609-8b3e-18570678c03b" containerID="b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2" exitCode=0
Sep 16 15:12:40.794613 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.794415    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb4b58ccc-z9886"
Sep 16 15:12:40.794613 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.794433    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb4b58ccc-z9886" event={"ID":"49d268f1-8000-4609-8b3e-18570678c03b","Type":"ContainerDied","Data":"b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2"}
Sep 16 15:12:40.794613 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.794484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb4b58ccc-z9886" event={"ID":"49d268f1-8000-4609-8b3e-18570678c03b","Type":"ContainerDied","Data":"1054dd95519961dd0e40ce6cb435ce659848ea5de3600ae96c49c6c8e63cc0dd"}
Sep 16 15:12:40.794613 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.794511    2558 scope.go:117] "RemoveContainer" containerID="b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2"
Sep 16 15:12:40.803293 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.803275    2558 scope.go:117] "RemoveContainer" containerID="b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2"
Sep 16 15:12:40.803547 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:12:40.803531    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2\": container with ID starting with b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2 not found: ID does not exist" containerID="b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2"
Sep 16 15:12:40.803590 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.803557    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2"} err="failed to get container status \"b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2\": rpc error: code = NotFound desc = could not find container \"b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2\": container with ID starting with b71213e24ec83962631aca00b6fa049354093e4bb3a560381dbbf67d2ad72fa2 not found: ID does not exist"
Sep 16 15:12:40.819554 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.819530    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-z9886"]
Sep 16 15:12:40.824292 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:40.824261    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-z9886"]
Sep 16 15:12:41.294106 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.294065    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49d268f1-8000-4609-8b3e-18570678c03b" path="/var/lib/kubelet/pods/49d268f1-8000-4609-8b3e-18570678c03b/volumes"
Sep 16 15:12:41.603812 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.603780    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-s8pbd"]
Sep 16 15:12:41.604118 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.604107    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d268f1-8000-4609-8b3e-18570678c03b" containerName="manager"
Sep 16 15:12:41.604187 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.604120    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d268f1-8000-4609-8b3e-18570678c03b" containerName="manager"
Sep 16 15:12:41.604237 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.604195    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d268f1-8000-4609-8b3e-18570678c03b" containerName="manager"
Sep 16 15:12:41.608492 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.608477    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:41.628293 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.628271    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 15:12:41.628426 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.628326    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 15:12:41.628550 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.628535    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tcknf\""
Sep 16 15:12:41.628623 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.628558    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 15:12:41.691914 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.691888    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcmzt\" (UniqueName: \"kubernetes.io/projected/155de48e-4864-4374-ab68-5bdff75f13a2-kube-api-access-mcmzt\") pod \"maas-controller-5cb4b58ccc-s8pbd\" (UID: \"155de48e-4864-4374-ab68-5bdff75f13a2\") " pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:41.692052 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.691956    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/155de48e-4864-4374-ab68-5bdff75f13a2-metrics-certs\") pod \"maas-controller-5cb4b58ccc-s8pbd\" (UID: \"155de48e-4864-4374-ab68-5bdff75f13a2\") " pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:41.692052 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.692008    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/155de48e-4864-4374-ab68-5bdff75f13a2-webhook-cert\") pod \"maas-controller-5cb4b58ccc-s8pbd\" (UID: \"155de48e-4864-4374-ab68-5bdff75f13a2\") " pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:41.696453 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.696429    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-s8pbd"]
Sep 16 15:12:41.792539 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.792503    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/155de48e-4864-4374-ab68-5bdff75f13a2-metrics-certs\") pod \"maas-controller-5cb4b58ccc-s8pbd\" (UID: \"155de48e-4864-4374-ab68-5bdff75f13a2\") " pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:41.792724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.792558    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/155de48e-4864-4374-ab68-5bdff75f13a2-webhook-cert\") pod \"maas-controller-5cb4b58ccc-s8pbd\" (UID: \"155de48e-4864-4374-ab68-5bdff75f13a2\") " pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:41.792724 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.792601    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcmzt\" (UniqueName: \"kubernetes.io/projected/155de48e-4864-4374-ab68-5bdff75f13a2-kube-api-access-mcmzt\") pod \"maas-controller-5cb4b58ccc-s8pbd\" (UID: \"155de48e-4864-4374-ab68-5bdff75f13a2\") " pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:41.795040 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.795016    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/155de48e-4864-4374-ab68-5bdff75f13a2-metrics-certs\") pod \"maas-controller-5cb4b58ccc-s8pbd\" (UID: \"155de48e-4864-4374-ab68-5bdff75f13a2\") " pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:41.795151 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.795051    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/155de48e-4864-4374-ab68-5bdff75f13a2-webhook-cert\") pod \"maas-controller-5cb4b58ccc-s8pbd\" (UID: \"155de48e-4864-4374-ab68-5bdff75f13a2\") " pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:41.801228 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.801203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcmzt\" (UniqueName: \"kubernetes.io/projected/155de48e-4864-4374-ab68-5bdff75f13a2-kube-api-access-mcmzt\") pod \"maas-controller-5cb4b58ccc-s8pbd\" (UID: \"155de48e-4864-4374-ab68-5bdff75f13a2\") " pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:41.917828 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:41.917735    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:42.193059 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:42.193033    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cb4b58ccc-s8pbd"]
Sep 16 15:12:42.195107 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:12:42.195078    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155de48e_4864_4374_ab68_5bdff75f13a2.slice/crio-229d411ccf5105a0c4b72dbd5f146734f40280270fac67344695a855525bf841 WatchSource:0}: Error finding container 229d411ccf5105a0c4b72dbd5f146734f40280270fac67344695a855525bf841: Status 404 returned error can't find the container with id 229d411ccf5105a0c4b72dbd5f146734f40280270fac67344695a855525bf841
Sep 16 15:12:42.806253 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:42.806215    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd" event={"ID":"155de48e-4864-4374-ab68-5bdff75f13a2","Type":"ContainerStarted","Data":"229d411ccf5105a0c4b72dbd5f146734f40280270fac67344695a855525bf841"}
Sep 16 15:12:43.812647 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:43.812614    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd" event={"ID":"155de48e-4864-4374-ab68-5bdff75f13a2","Type":"ContainerStarted","Data":"bbc916618aac87f34880d2046d0753c3c9507bf794942ab068140aa73aef4a36"}
Sep 16 15:12:43.813077 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:43.812742    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:12:43.835338 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:43.835290    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd" podStartSLOduration=2.002269611 podStartE2EDuration="2.835276636s" totalImagesPullingTime="833.007025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:12:43.833998266 +0000 UTC m=+2240.988076552" watchObservedRunningTime="2026-09-16 15:12:43.835276636 +0000 UTC m=+2240.989354963"
Sep 16 15:12:54.820161 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:12:54.820084    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cb4b58ccc-s8pbd"
Sep 16 15:15:00.146893 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:00.146849    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826195-npc26"]
Sep 16 15:15:00.150214 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:00.150198    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826195-npc26"
Sep 16 15:15:00.154886 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:00.154860    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5dqq2\""
Sep 16 15:15:00.159373 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:00.159354    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826195-npc26"
Sep 16 15:15:00.171194 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:00.171160    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826195-npc26"]
Sep 16 15:15:00.437349 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:00.437325    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826195-npc26"]
Sep 16 15:15:00.439826 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:15:00.439801    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d18b743_25ae_4f5b_8600_a23b95d518ba.slice/crio-7990d78d05594d321960468c9dc4d27374c0fb7d6457a775ac0fa496874087fd WatchSource:0}: Error finding container 7990d78d05594d321960468c9dc4d27374c0fb7d6457a775ac0fa496874087fd: Status 404 returned error can't find the container with id 7990d78d05594d321960468c9dc4d27374c0fb7d6457a775ac0fa496874087fd
Sep 16 15:15:00.459506 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:00.459472    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826195-npc26" event={"ID":"5d18b743-25ae-4f5b-8600-a23b95d518ba","Type":"ContainerStarted","Data":"7990d78d05594d321960468c9dc4d27374c0fb7d6457a775ac0fa496874087fd"}
Sep 16 15:15:01.466321 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:01.466292    2558 generic.go:356] "Generic (PLEG): container finished" podID="5d18b743-25ae-4f5b-8600-a23b95d518ba" containerID="63e9c5ce4e6587d25201f4dce52885f973b178bc2dceedb348bba088236c449f" exitCode=0
Sep 16 15:15:01.466686 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:01.466337    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826195-npc26" event={"ID":"5d18b743-25ae-4f5b-8600-a23b95d518ba","Type":"ContainerDied","Data":"63e9c5ce4e6587d25201f4dce52885f973b178bc2dceedb348bba088236c449f"}
Sep 16 15:15:02.598938 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:02.598914    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826195-npc26"
Sep 16 15:15:03.477143 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:03.477114    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826195-npc26"
Sep 16 15:15:03.477322 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:03.477112    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826195-npc26" event={"ID":"5d18b743-25ae-4f5b-8600-a23b95d518ba","Type":"ContainerDied","Data":"7990d78d05594d321960468c9dc4d27374c0fb7d6457a775ac0fa496874087fd"}
Sep 16 15:15:03.477322 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:03.477244    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7990d78d05594d321960468c9dc4d27374c0fb7d6457a775ac0fa496874087fd"
Sep 16 15:15:23.411651 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:23.411623    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 15:15:23.418960 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:23.418937    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vskvj_bbc64f67-7703-4482-9ebe-fe6f420c8bc0/ovn-acl-logging/0.log"
Sep 16 15:15:23.903726 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:23.903698    2558 scope.go:117] "RemoveContainer" containerID="91e709fc962059cc051e68c54d0447d2dae3a5387e955884df8b798f13fe5663"
Sep 16 15:15:23.911607 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:23.911589    2558 scope.go:117] "RemoveContainer" containerID="ac27aa85902d2bf6fc91439b104c22f25acf1c517e67106af1ec084321ac3205"
Sep 16 15:15:44.827723 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.827694    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"]
Sep 16 15:15:44.828213 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.828065    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d18b743-25ae-4f5b-8600-a23b95d518ba" containerName="cleanup"
Sep 16 15:15:44.828213 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.828079    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d18b743-25ae-4f5b-8600-a23b95d518ba" containerName="cleanup"
Sep 16 15:15:44.828213 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.828177    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d18b743-25ae-4f5b-8600-a23b95d518ba" containerName="cleanup"
Sep 16 15:15:44.831398 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.831379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.834509 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.834489    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c60debce-kserve-self-signed-certs\""
Sep 16 15:15:44.845226 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.845207    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"]
Sep 16 15:15:44.866680 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.866652    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tmp-dir\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.866807 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.866710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-home\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.866807 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.866757    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-dshm\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.866807 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.866790    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghfkh\" (UniqueName: \"kubernetes.io/projected/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-kube-api-access-ghfkh\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.867005 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.866815    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tls-certs\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.867005 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.866832    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-model-cache\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.967810 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.967782    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-home\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.967976 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.967829    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-dshm\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.967976 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.967854    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghfkh\" (UniqueName: \"kubernetes.io/projected/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-kube-api-access-ghfkh\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.967976 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.967873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tls-certs\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.967976 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.967892    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-model-cache\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.967976 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.967944    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tmp-dir\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.968292 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.968229    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-home\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.968292 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.968281    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tmp-dir\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.968386 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.968366    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-model-cache\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.970206 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.970184    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-dshm\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.970346 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.970329    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tls-certs\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:44.976958 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:44.976937    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghfkh\" (UniqueName: \"kubernetes.io/projected/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-kube-api-access-ghfkh\") pod \"e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") " pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:45.140921 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:45.140836    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:45.437673 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:45.437649    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"]
Sep 16 15:15:45.439792 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:15:45.439766    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb5b3b40_133b_4346_ae6f_0f26ed4fedc3.slice/crio-657bf3f1419a3b04b75a8d79a81b50a99b280c1dad0f3efd1e66ffcbfb798963 WatchSource:0}: Error finding container 657bf3f1419a3b04b75a8d79a81b50a99b280c1dad0f3efd1e66ffcbfb798963: Status 404 returned error can't find the container with id 657bf3f1419a3b04b75a8d79a81b50a99b280c1dad0f3efd1e66ffcbfb798963
Sep 16 15:15:45.678388 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:45.678356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t" event={"ID":"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3","Type":"ContainerStarted","Data":"3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a"}
Sep 16 15:15:45.678566 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:45.678400    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t" event={"ID":"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3","Type":"ContainerStarted","Data":"657bf3f1419a3b04b75a8d79a81b50a99b280c1dad0f3efd1e66ffcbfb798963"}
Sep 16 15:15:45.719642 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:45.719544    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t" podStartSLOduration=1.71952862 podStartE2EDuration="1.71952862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 15:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:15:45.718975046 +0000 UTC m=+2422.873053333" watchObservedRunningTime="2026-09-16 15:15:45.71952862 +0000 UTC m=+2422.873606906"
Sep 16 15:15:55.141927 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:55.141889    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:55.142464 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:55.141971    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:55.154321 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:55.154301    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:15:55.736431 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:15:55.736403    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:16:23.931790 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:23.931755    2558 scope.go:117] "RemoveContainer" containerID="1ae963edf4d124ffc5703aa0d8e21afa630da932538899a46910ca2ce1814758"
Sep 16 15:16:25.263324 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:25.263291    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"]
Sep 16 15:16:40.282033 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.281972    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t" podUID="eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" containerName="main" containerID="cri-o://3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a" gracePeriod=45
Sep 16 15:16:40.520545 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.520523    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:16:40.566707 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.566682    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-home\") pod \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") "
Sep 16 15:16:40.566841 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.566712    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tmp-dir\") pod \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") "
Sep 16 15:16:40.566841 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.566745    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghfkh\" (UniqueName: \"kubernetes.io/projected/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-kube-api-access-ghfkh\") pod \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") "
Sep 16 15:16:40.566841 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.566782    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-model-cache\") pod \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") "
Sep 16 15:16:40.566841 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.566821    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-dshm\") pod \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") "
Sep 16 15:16:40.567005 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.566870    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tls-certs\") pod \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\" (UID: \"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3\") "
Sep 16 15:16:40.567060 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.566995    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-home" (OuterVolumeSpecName: "home") pod "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" (UID: "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:16:40.567060 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.567002    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" (UID: "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:16:40.567191 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.567079    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" (UID: "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:16:40.567259 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.567200    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-model-cache\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:16:40.567259 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.567211    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-home\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:16:40.567259 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.567219    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tmp-dir\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:16:40.569033 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.569007    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-kube-api-access-ghfkh" (OuterVolumeSpecName: "kube-api-access-ghfkh") pod "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" (UID: "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3"). InnerVolumeSpecName "kube-api-access-ghfkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 15:16:40.569153 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.569035    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" (UID: "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 15:16:40.569153 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.569104    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-dshm" (OuterVolumeSpecName: "dshm") pod "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" (UID: "eb5b3b40-133b-4346-ae6f-0f26ed4fedc3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 15:16:40.667714 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.667675    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghfkh\" (UniqueName: \"kubernetes.io/projected/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-kube-api-access-ghfkh\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:16:40.667714 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.667707    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-dshm\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:16:40.667714 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.667720    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3-tls-certs\") on node \"ip-10-0-128-123.ec2.internal\" DevicePath \"\""
Sep 16 15:16:40.932223 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.932113    2558 generic.go:356] "Generic (PLEG): container finished" podID="eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" containerID="3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a" exitCode=0
Sep 16 15:16:40.932223 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.932188    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"
Sep 16 15:16:40.932223 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.932199    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t" event={"ID":"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3","Type":"ContainerDied","Data":"3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a"}
Sep 16 15:16:40.932442 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.932237    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t" event={"ID":"eb5b3b40-133b-4346-ae6f-0f26ed4fedc3","Type":"ContainerDied","Data":"657bf3f1419a3b04b75a8d79a81b50a99b280c1dad0f3efd1e66ffcbfb798963"}
Sep 16 15:16:40.932442 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.932254    2558 scope.go:117] "RemoveContainer" containerID="3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a"
Sep 16 15:16:40.941025 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.941008    2558 scope.go:117] "RemoveContainer" containerID="3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a"
Sep 16 15:16:40.941300 ip-10-0-128-123 kubenswrapper[2558]: E0916 15:16:40.941280    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a\": container with ID starting with 3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a not found: ID does not exist" containerID="3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a"
Sep 16 15:16:40.941371 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.941308    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a"} err="failed to get container status \"3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a\": rpc error: code = NotFound desc = could not find container \"3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a\": container with ID starting with 3a81c61a075ef7a9dac16ba007fa4554452172cccb1ad61f95cc7bee8b7b089a not found: ID does not exist"
Sep 16 15:16:40.955574 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.955551    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"]
Sep 16 15:16:40.957122 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:40.957105    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c60debce-kserve-88bd6cbbc-hkl6t"]
Sep 16 15:16:41.295325 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:16:41.295252    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" path="/var/lib/kubelet/pods/eb5b3b40-133b-4346-ae6f-0f26ed4fedc3/volumes"
Sep 16 15:17:17.002964 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:17.002875    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6998bc65dd-7n6g4_9adff22f-71ea-4168-b8af-c2a3d22ab2d9/authorino/0.log"
Sep 16 15:17:22.673287 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:22.673255    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cb4b58ccc-s8pbd_155de48e-4864-4374-ab68-5bdff75f13a2/manager/0.log"
Sep 16 15:17:23.034145 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:23.034068    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-846667779d-6p5jp_817c7241-feea-40c9-a095-af6e0ca69bb7/copy-manifests/0.log"
Sep 16 15:17:23.041196 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:23.041175    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-846667779d-6p5jp_817c7241-feea-40c9-a095-af6e0ca69bb7/manager/0.log"
Sep 16 15:17:23.833412 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:23.833381    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826180-sr7t5_c3cde76d-cc6e-4616-86c1-f7d8664f2e0c/cleanup/0.log"
Sep 16 15:17:23.942909 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:23.942870    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826195-npc26_5d18b743-25ae-4f5b-8600-a23b95d518ba/cleanup/0.log"
Sep 16 15:17:23.952226 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:23.952203    2558 scope.go:117] "RemoveContainer" containerID="0260f844c3cf1a5bb8cc8fcd7a8d259f787cdc8398b3a17edc0da883aa0b15eb"
Sep 16 15:17:23.959801 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:23.959782    2558 scope.go:117] "RemoveContainer" containerID="c1d3d6421a2bf47a55ff9793b1dc408de1fe3257746b633812928ff9b7b2572d"
Sep 16 15:17:24.055346 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:24.055323    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-jcfz4_c9cfbe47-6d5b-420e-abb0-c3cc59a6b69c/postgres/0.log"
Sep 16 15:17:24.856412 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:24.856385    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6998bc65dd-7n6g4_9adff22f-71ea-4168-b8af-c2a3d22ab2d9/authorino/0.log"
Sep 16 15:17:25.189129 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:25.189053    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-qxh5s_7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99/kuadrant-console-plugin/0.log"
Sep 16 15:17:26.126234 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:26.126207    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d79898bc9-j4nvg_69122b7c-a0c3-4304-b291-cd6dbda74797/kube-auth-proxy/0.log"
Sep 16 15:17:26.464816 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:26.464738    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2zpjm_8f7fbf47-02a6-4b60-b02f-4ca4fb581dba/payload-pre-processing/0.log"
Sep 16 15:17:26.588928 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:26.588902    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4z9lb_4b0b07a0-1590-46ec-bc89-937aeeb72f6f/payload-processing/0.log"
Sep 16 15:17:27.152155 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:27.152126    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-9qxcq_09827ed2-f19d-4840-988d-441e9530b3a1/main/0.log"
Sep 16 15:17:27.585407 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:27.585381    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-f48bt_90a50414-7ce6-4f89-a007-d09496a32bd3/main/0.log"
Sep 16 15:17:27.693648 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:27.693615    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-tht5l_f0b35c03-838c-4d10-926f-64d1420946df/main/0.log"
Sep 16 15:17:36.307298 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:36.307252    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r9l6v_82cdd79b-96ab-48dd-a670-0d59194a7992/global-pull-secret-syncer/0.log"
Sep 16 15:17:36.483949 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:36.483919    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j8tbw_e1641169-72fe-4cfb-b569-60d355a4f551/konnectivity-agent/0.log"
Sep 16 15:17:36.506789 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:36.506766    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-123.ec2.internal_e99847c9ed9bfebb0b4989bf767e4cc0/haproxy/0.log"
Sep 16 15:17:41.124636 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:41.124602    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6998bc65dd-7n6g4_9adff22f-71ea-4168-b8af-c2a3d22ab2d9/authorino/0.log"
Sep 16 15:17:41.236075 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:41.236051    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-qxh5s_7fed9b26-abf0-4d2a-8cbc-62fc3cbcad99/kuadrant-console-plugin/0.log"
Sep 16 15:17:42.816345 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:42.816314    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fe6edb56-54eb-4e31-a385-98ea97e6feb6/alertmanager/0.log"
Sep 16 15:17:42.847202 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:42.847160    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fe6edb56-54eb-4e31-a385-98ea97e6feb6/config-reloader/0.log"
Sep 16 15:17:42.875981 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:42.875956    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fe6edb56-54eb-4e31-a385-98ea97e6feb6/kube-rbac-proxy-web/0.log"
Sep 16 15:17:42.907384 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:42.907316    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fe6edb56-54eb-4e31-a385-98ea97e6feb6/kube-rbac-proxy/0.log"
Sep 16 15:17:42.935210 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:42.935187    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fe6edb56-54eb-4e31-a385-98ea97e6feb6/kube-rbac-proxy-metric/0.log"
Sep 16 15:17:42.960266 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:42.960244    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fe6edb56-54eb-4e31-a385-98ea97e6feb6/prom-label-proxy/0.log"
Sep 16 15:17:42.986751 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:42.986729    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fe6edb56-54eb-4e31-a385-98ea97e6feb6/init-config-reloader/0.log"
Sep 16 15:17:43.178314 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.178245    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-684fbdf947-xtm97_fcc44efc-a893-4e75-b805-765ab2aca444/metrics-server/0.log"
Sep 16 15:17:43.477106 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.477029    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f2jms_b94269a4-3929-424a-bbb8-c50f384d1754/node-exporter/0.log"
Sep 16 15:17:43.505110 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.505074    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f2jms_b94269a4-3929-424a-bbb8-c50f384d1754/kube-rbac-proxy/0.log"
Sep 16 15:17:43.533712 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.533685    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f2jms_b94269a4-3929-424a-bbb8-c50f384d1754/init-textfile/0.log"
Sep 16 15:17:43.684870 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.684838    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a9b3d956-68a6-4e23-b10d-4aef5c396f70/prometheus/0.log"
Sep 16 15:17:43.714293 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.714268    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a9b3d956-68a6-4e23-b10d-4aef5c396f70/config-reloader/0.log"
Sep 16 15:17:43.740206 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.740116    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a9b3d956-68a6-4e23-b10d-4aef5c396f70/thanos-sidecar/0.log"
Sep 16 15:17:43.782269 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.782244    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a9b3d956-68a6-4e23-b10d-4aef5c396f70/kube-rbac-proxy-web/0.log"
Sep 16 15:17:43.809387 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.809360    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a9b3d956-68a6-4e23-b10d-4aef5c396f70/kube-rbac-proxy/0.log"
Sep 16 15:17:43.851143 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.851107    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a9b3d956-68a6-4e23-b10d-4aef5c396f70/kube-rbac-proxy-thanos/0.log"
Sep 16 15:17:43.882359 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.882336    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a9b3d956-68a6-4e23-b10d-4aef5c396f70/init-config-reloader/0.log"
Sep 16 15:17:43.919443 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.919403    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-fqk9w_1d8253c9-5d09-4990-aedc-7072f4b53d1c/prometheus-operator/0.log"
Sep 16 15:17:43.942605 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.942578    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-fqk9w_1d8253c9-5d09-4990-aedc-7072f4b53d1c/kube-rbac-proxy/0.log"
Sep 16 15:17:43.973528 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:43.973498    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-hcw2d_aeace53c-a99e-40b4-8de7-743f53f5e872/prometheus-operator-admission-webhook/0.log"
Sep 16 15:17:46.356261 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.356229    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"]
Sep 16 15:17:46.356644 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.356571    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" containerName="main"
Sep 16 15:17:46.356644 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.356582    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" containerName="main"
Sep 16 15:17:46.356718 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.356650    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb5b3b40-133b-4346-ae6f-0f26ed4fedc3" containerName="main"
Sep 16 15:17:46.359841 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.359821    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.364196 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.364159    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7m9cp\"/\"kube-root-ca.crt\""
Sep 16 15:17:46.364986 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.364964    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7m9cp\"/\"default-dockercfg-kx9vz\""
Sep 16 15:17:46.365100 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.365005    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7m9cp\"/\"openshift-service-ca.crt\""
Sep 16 15:17:46.377608 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.377584    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"]
Sep 16 15:17:46.441129 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.441093    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cgbj\" (UniqueName: \"kubernetes.io/projected/27161dfc-5e4c-48b3-900c-936d0022d5c4-kube-api-access-9cgbj\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.441318 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.441148    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-lib-modules\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.441318 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.441271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-sys\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.441318 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.441315    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-proc\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.441428 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.441347    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-podres\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.541898 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.541864    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cgbj\" (UniqueName: \"kubernetes.io/projected/27161dfc-5e4c-48b3-900c-936d0022d5c4-kube-api-access-9cgbj\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.542048 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.541919    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-lib-modules\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.542048 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.541953    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-sys\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.542048 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.541978    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-proc\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.542247 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.542053    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-sys\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.542247 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.542090    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-proc\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.542247 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.542122    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-podres\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.542247 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.542133    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-lib-modules\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.542247 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.542208    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27161dfc-5e4c-48b3-900c-936d0022d5c4-podres\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.550740 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.550719    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cgbj\" (UniqueName: \"kubernetes.io/projected/27161dfc-5e4c-48b3-900c-936d0022d5c4-kube-api-access-9cgbj\") pod \"perf-node-gather-daemonset-zv5bm\" (UID: \"27161dfc-5e4c-48b3-900c-936d0022d5c4\") " pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.669549 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.669443    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:46.939156 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:46.939131    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"]
Sep 16 15:17:46.941837 ip-10-0-128-123 kubenswrapper[2558]: W0916 15:17:46.941810    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod27161dfc_5e4c_48b3_900c_936d0022d5c4.slice/crio-e8a55274150af0deabc0a9c32aeb49a92ea9b90790eb6de69c4410fc04dd34a8 WatchSource:0}: Error finding container e8a55274150af0deabc0a9c32aeb49a92ea9b90790eb6de69c4410fc04dd34a8: Status 404 returned error can't find the container with id e8a55274150af0deabc0a9c32aeb49a92ea9b90790eb6de69c4410fc04dd34a8
Sep 16 15:17:47.252021 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:47.251936    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm" event={"ID":"27161dfc-5e4c-48b3-900c-936d0022d5c4","Type":"ContainerStarted","Data":"f45c19aebda7fc03c4f55951a22d67fb84ccc0fbc54cb83c0172d60af98a7b63"}
Sep 16 15:17:47.252021 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:47.251987    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm" event={"ID":"27161dfc-5e4c-48b3-900c-936d0022d5c4","Type":"ContainerStarted","Data":"e8a55274150af0deabc0a9c32aeb49a92ea9b90790eb6de69c4410fc04dd34a8"}
Sep 16 15:17:47.252231 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:47.252071    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:47.270567 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:47.270523    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm" podStartSLOduration=1.270508901 podStartE2EDuration="1.270508901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 15:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 15:17:47.269449128 +0000 UTC m=+2544.423527407" watchObservedRunningTime="2026-09-16 15:17:47.270508901 +0000 UTC m=+2544.424587203"
Sep 16 15:17:48.751603 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:48.751579    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wqn5h_a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985/dns/0.log"
Sep 16 15:17:48.783953 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:48.783927    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wqn5h_a85d3d0e-b3c1-40cd-90c5-6cba2ebaf985/kube-rbac-proxy/0.log"
Sep 16 15:17:48.843781 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:48.843745    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jzrt4_0fa62dc0-5f61-4349-b7c6-9d4242503566/dns-node-resolver/0.log"
Sep 16 15:17:49.464879 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:49.464851    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vpphh_9be55def-d862-4674-864b-04119c40ceeb/node-ca/0.log"
Sep 16 15:17:50.861134 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:50.861096    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d79898bc9-j4nvg_69122b7c-a0c3-4304-b291-cd6dbda74797/kube-auth-proxy/0.log"
Sep 16 15:17:50.975441 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:50.975410    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2zpjm_8f7fbf47-02a6-4b60-b02f-4ca4fb581dba/payload-pre-processing/0.log"
Sep 16 15:17:51.067287 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:51.067257    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4z9lb_4b0b07a0-1590-46ec-bc89-937aeeb72f6f/payload-processing/0.log"
Sep 16 15:17:52.079453 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:52.079428    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zl54r_dadc893b-cc06-4432-ad74-8a14046e81fd/serve-healthcheck-canary/0.log"
Sep 16 15:17:52.623994 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:52.623969    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-69hhl_3f36b8b3-e873-4e80-8755-76a5fcab21c3/kube-rbac-proxy/0.log"
Sep 16 15:17:52.646581 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:52.646551    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-69hhl_3f36b8b3-e873-4e80-8755-76a5fcab21c3/exporter/0.log"
Sep 16 15:17:52.669419 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:52.669398    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-69hhl_3f36b8b3-e873-4e80-8755-76a5fcab21c3/extractor/0.log"
Sep 16 15:17:53.266129 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:53.266102    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7m9cp/perf-node-gather-daemonset-zv5bm"
Sep 16 15:17:55.640890 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:55.640863    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5cb4b58ccc-s8pbd_155de48e-4864-4374-ab68-5bdff75f13a2/manager/0.log"
Sep 16 15:17:55.876798 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:55.876767    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-846667779d-6p5jp_817c7241-feea-40c9-a095-af6e0ca69bb7/manager/0.log"
Sep 16 15:17:55.924045 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:55.924011    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-846667779d-6p5jp_817c7241-feea-40c9-a095-af6e0ca69bb7/copy-manifests/0.log"
Sep 16 15:17:56.825077 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:56.825049    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d79898bc9-j4nvg_69122b7c-a0c3-4304-b291-cd6dbda74797/kube-auth-proxy/0.log"
Sep 16 15:17:56.885038 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:56.885007    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2zpjm_8f7fbf47-02a6-4b60-b02f-4ca4fb581dba/payload-pre-processing/0.log"
Sep 16 15:17:56.905940 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:56.905912    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4z9lb_4b0b07a0-1590-46ec-bc89-937aeeb72f6f/payload-processing/0.log"
Sep 16 15:17:57.451742 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:57.451704    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8787d76cb-lt6k8_f2b65df2-60ea-42f6-b57e-913ada173141/manager/0.log"
Sep 16 15:17:57.515753 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:57.515721    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-mq6t6_c6d26148-68b6-4557-94fa-1c4f60ac8bdb/openshift-lws-operator/0.log"
Sep 16 15:17:58.948359 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:58.948326    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d79898bc9-j4nvg_69122b7c-a0c3-4304-b291-cd6dbda74797/kube-auth-proxy/0.log"
Sep 16 15:17:59.017209 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:59.017157    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2zpjm_8f7fbf47-02a6-4b60-b02f-4ca4fb581dba/payload-pre-processing/0.log"
Sep 16 15:17:59.038953 ip-10-0-128-123 kubenswrapper[2558]: I0916 15:17:59.038929    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-4z9lb_4b0b07a0-1590-46ec-bc89-937aeeb72f6f/payload-processing/0.log"