Sep 09 23:48:39.635309 ip-10-0-139-211 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 23:48:40.061873 ip-10-0-139-211 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 23:48:40.061873 ip-10-0-139-211 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 23:48:40.061873 ip-10-0-139-211 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 23:48:40.061873 ip-10-0-139-211 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 23:48:40.061873 ip-10-0-139-211 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 23:48:40.062917 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.062835    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 23:48:40.068816 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068799    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:40.068816 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068814    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:40.068816 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068818    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068822    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068825    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068828    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068831    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068834    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068838    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068841    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068844    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068847    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068850    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068852    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068855    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068858    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068861    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068866    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068869    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068872    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068874    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068877    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:40.068911 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068880    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068883    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068885    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068888    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068892    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068896    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068900    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068902    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068905    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068908    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068913    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068917    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068921    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068924    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068927    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068930    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068933    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068936    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068939    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068942    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:40.069408 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068945    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068948    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068952    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068955    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068958    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068961    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068964    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068967    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068970    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068972    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068975    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068978    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068980    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068983    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068987    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068990    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068992    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068995    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.068998    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:40.069908 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069001    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069005    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069008    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069011    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069014    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069017    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069020    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069022    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069026    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069028    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069031    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069034    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069036    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069039    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069042    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069044    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069047    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069050    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069064    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069068    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:40.070390 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069071    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069074    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069076    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069079    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069081    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069084    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069088    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069091    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069094    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069096    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069099    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069102    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069105    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069107    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069110    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069113    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069116    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069118    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069784    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:40.070873 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069790    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069793    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069796    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069799    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069801    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069804    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069807    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069810    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069812    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069815    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069817    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069820    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069823    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069826    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069829    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069831    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069834    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069837    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069840    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069842    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:40.071366 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069846    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069849    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069851    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069854    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069857    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069859    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069862    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069866    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069870    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069873    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069875    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069878    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069882    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069886    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069888    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069891    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069894    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069899    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069902    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:40.071870 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069904    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069908    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069910    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069913    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069916    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069919    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069922    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069925    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069928    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069930    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069933    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069936    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069939    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069941    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069945    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069947    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069950    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069952    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069955    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:40.072368 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069957    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069960    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069962    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069966    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069970    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069973    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069976    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069979    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069981    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069985    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069988    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069990    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069993    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069995    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.069998    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070000    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070003    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070005    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070008    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070010    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:40.072836 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070013    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070016    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070019    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070022    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070024    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070027    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070030    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070033    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070035    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070038    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070040    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070043    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070045    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070048    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070050    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070068    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070071    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070073    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070076    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070079    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:40.073346 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070442    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070447    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070451    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070454    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070456    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070459    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070462    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070464    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070467    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070470    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070473    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070475    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070478    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070484    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070486    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070489    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070492    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070495    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070497    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:40.073865 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070500    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070502    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070505    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070507    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070510    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070513    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070515    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070518    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070521    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070524    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070526    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070529    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070532    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070535    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070537    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070541    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070544    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070546    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070549    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:40.074348 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070552    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070556    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070560    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070563    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070565    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070570    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070573    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070579    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070583    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070586    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070589    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070592    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070595    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070598    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070600    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070603    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070605    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070608    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070610    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:40.074820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070613    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070615    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070620    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070622    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070625    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070627    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070631    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070634    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070636    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070639    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070642    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070644    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070647    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070649    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070652    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070656    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070659    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070662    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070665    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:40.075351 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070668    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070673    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070676    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070680    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070683    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070685    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070688    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070690    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070693    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070695    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070698    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070701    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070703    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070706    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070708    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070712    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070715    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070717    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070720    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070722    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:40.075829 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070725    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070727    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.070730    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070801    2556 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070832    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070838    2556 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070842    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070846    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070850    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070854    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070858    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070861    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070864    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070868    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070871    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070875    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070879    2556 flags.go:64] FLAG: --cgroup-root=""
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070882    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070885    2556 flags.go:64] FLAG: --client-ca-file=""
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070888    2556 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070891    2556 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070895    2556 flags.go:64] FLAG: --cluster-domain=""
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070898    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070901    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 23:48:40.076374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070905    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070908    2556 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070912    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070917    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070920    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070923    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070926    2556 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070929    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070932    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070935    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070938    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070942    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070945    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070948    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070951    2556 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070954    2556 flags.go:64] FLAG: --enable-server="true"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070957    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070961    2556 flags.go:64] FLAG: --event-burst="100"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070964    2556 flags.go:64] FLAG: --event-qps="50"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070966    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070969    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070973    2556 flags.go:64] FLAG: --eviction-hard=""
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070976    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070979    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070982    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 23:48:40.076962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070985    2556 flags.go:64] FLAG: --eviction-soft=""
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070988    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070991    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070994    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070997    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.070999    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071002    2556 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071005    2556 flags.go:64] FLAG: --feature-gates=""
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071009    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071011    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071016    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071019    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071022    2556 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071025    2556 flags.go:64] FLAG: --help="false"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071028    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071031    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071034    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071037    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071040    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071043    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071046    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071049    2556 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071065    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 23:48:40.077588 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071069    2556 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071072    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071075    2556 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071078    2556 flags.go:64] FLAG: --kube-reserved=""
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071080    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071084    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071087    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071090    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071093    2556 flags.go:64] FLAG: --lock-file=""
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071096    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071099    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071102    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071107    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071110    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071113    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071116    2556 flags.go:64] FLAG: --logging-format="text"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071118    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071122    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071124    2556 flags.go:64] FLAG: --manifest-url=""
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071129    2556 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071133    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071136    2556 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071140    2556 flags.go:64] FLAG: --max-pods="110"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071143    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071146    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 23:48:40.078279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071149    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071152    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071155    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071158    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071161    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071168    2556 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071171    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071174    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071177    2556 flags.go:64] FLAG: --pod-cidr=""
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071180    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071186    2556 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071189    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071193    2556 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071196    2556 flags.go:64] FLAG: --port="10250"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071200    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071203    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-026ae125ad8b2fe13"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071206    2556 flags.go:64] FLAG: --qos-reserved=""
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071209    2556 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071213    2556 flags.go:64] FLAG: --register-node="true"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071216    2556 flags.go:64] FLAG: --register-with-taints=""
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071219    2556 flags.go:64] FLAG: --registry-burst="10"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071222    2556 flags.go:64] FLAG: --registry-qps="5"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071225    2556 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071228    2556 flags.go:64] FLAG: --reserved-memory=""
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071231    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 23:48:40.079145 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071234    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071238    2556 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071241    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071244    2556 flags.go:64] FLAG: --runonce="false"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071247    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071250    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071253    2556 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071256    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071259    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071262    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071265    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071269    2556 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071272    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071275    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071277    2556 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071281    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071284    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071287    2556 flags.go:64] FLAG: --system-cgroups=""
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071290    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071295    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071298    2556 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071301    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071305    2556 flags.go:64] FLAG: --tls-min-version=""
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071308    2556 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071311    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 23:48:40.080254 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071314    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071317    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071320    2556 flags.go:64] FLAG: --v="2"
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071324    2556 flags.go:64] FLAG: --version="false"
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071328    2556 flags.go:64] FLAG: --vmodule=""
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071331    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.071335    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071454    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071459    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071462    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071465    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071469    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071472    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071474    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071477    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071480    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071482    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071485    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071488    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071491    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071493    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071495    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:40.081315 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071498    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071501    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071503    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071506    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071509    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071512    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071515    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071518    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071520    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071523    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071526    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071528    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071531    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071534    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071537    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071539    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071542    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071544    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071547    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:40.081853 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071550    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071553    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071555    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071558    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071561    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071565    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071569    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071572    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071575    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071578    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071581    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071583    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071586    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071589    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071591    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071594    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071597    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071601    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071603    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071606    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:40.082616 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071609    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071611    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071615    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071617    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071620    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071622    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071625    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071628    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071630    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071633    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071635    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071638    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071641    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071643    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071646    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071648    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071651    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071655    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071659    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:40.083494 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071662    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071665    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071668    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071671    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071674    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071677    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071680    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071682    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071685    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071687    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071691    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071694    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071697    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071700    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071704    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071707    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071709    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071712    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071714    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:40.083971 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071725    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071729    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071731    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071734    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071737    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071739    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.071742    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.072601    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.080938    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.080956    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.081979    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.081999    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082101    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082110    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082116    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:40.084677 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082122    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082126    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082131    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082136    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082140    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082145    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082149    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082154    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082158    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082162    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082167    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082171    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082175    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082179    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082184    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082188    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082192    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082196    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082200    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082204    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:40.085347 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082208    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082213    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082217    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082221    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082225    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082229    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082236    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082240    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082245    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082249    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082254    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082260    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082266    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082270    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082274    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082279    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082284    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082289    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082293    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:40.085990 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082298    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082303    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082307    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082312    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082316    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082320    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082324    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082331    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082336    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082342    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082346    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082351    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082356    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082361    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082365    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082370    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082374    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082378    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082382    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:40.086668 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082387    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082392    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082397    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082401    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082404    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082409    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082413    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082418    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082422    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082426    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082430    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082435    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082439    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082443    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082447    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082451    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082455    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082459    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082465    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082472    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:40.087167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082476    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082481    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082486    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082489    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082493    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082498    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082502    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082506    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082510    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082514    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082519    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082523    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082528    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082532    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082537    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082541    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082545    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:40.087719 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082549    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.082557    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082731    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082740    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082745    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082750    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082754    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082759    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082765    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082773    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082778    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082783    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082790    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082794    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 23:48:40.088160 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082798    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082803    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082807    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082811    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082815    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082819    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082824    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082828    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082832    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082836    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082841    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082845    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082851    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082857    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082862    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082867    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082873    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082878    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082883    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 23:48:40.088529 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082887    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082892    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082896    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082900    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082904    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082909    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082913    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082917    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082921    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082925    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082929    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082933    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082937    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082942    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082946    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082950    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082954    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082958    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082963    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082967    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 23:48:40.089037 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082971    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082975    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082979    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082984    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082988    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082992    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.082996    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083000    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083004    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083008    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083013    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083017    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083021    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083025    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083029    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083033    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083037    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083041    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083046    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083050    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083075    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 23:48:40.089626 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083079    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083083    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083087    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083091    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083095    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083099    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083103    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083108    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083112    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083116    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083120    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083124    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083128    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083132    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083137    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083141    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083145    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083149    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083154    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 23:48:40.090193 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083158    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083162    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083166    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083173    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083177    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083182    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083186    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:40.083190    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.083198    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.083385    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.086269    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.087508    2556 server.go:1019] "Starting client certificate rotation"
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.087612    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 23:48:40.090967 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.087651    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 23:48:40.091541 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.090788    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 23:48:40.095162 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.095133    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 23:48:40.103445 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.103427    2556 log.go:25] "Validated CRI v1 runtime API"
Sep 09 23:48:40.107482 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.107462    2556 log.go:25] "Validated CRI v1 image API"
Sep 09 23:48:40.108601 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.108587    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 23:48:40.113365 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.113342    2556 fs.go:135] Filesystem UUIDs: map[3b56ea11-094e-45fc-b046-43675ca49791:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ca37e479-954d-4555-9b60-259c40429ce3:/dev/nvme0n1p3]
Sep 09 23:48:40.113426 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.113366    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 23:48:40.120191 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.120078    2556 manager.go:216] Machine: {Timestamp:2026-09-09 23:48:40.118592723 +0000 UTC m=+0.369728761 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101820 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec208736bbaf8124d53525e6b5043940 SystemUUID:ec208736-bbaf-8124-d535-25e6b5043940 BootID:ed31a321-e8b2-48ad-b9cf-2b71703cdff1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:c6:44:72:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:c6:44:72:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:ea:04:9b:2f:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 23:48:40.120191 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.120190    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 23:48:40.120319 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.120307    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 23:48:40.123282 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.123264    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 23:48:40.123652 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.123632    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 23:48:40.123795 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.123657    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-211.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 23:48:40.124516 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.124506    2556 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 23:48:40.124549 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.124519    2556 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 23:48:40.124549 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.124527    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 23:48:40.126131 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.126118    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 23:48:40.128095 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.128085    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 23:48:40.132597 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.132585    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 23:48:40.132708 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.132698    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 23:48:40.132740 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.132728    2556 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 23:48:40.132740 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.132738    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 23:48:40.132803 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.132751    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 23:48:40.132803 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.132763    2556 kubelet.go:398] "Adding apiserver pod source"
Sep 09 23:48:40.132803 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.132771    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 23:48:40.133138 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.133122    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 23:48:40.133203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.133151    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 23:48:40.135452 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.135435    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 23:48:40.138261 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.138249    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 23:48:40.138309 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.138279    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 23:48:40.138510 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.138492    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 23:48:40.138510 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.138510    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 23:48:40.138567 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.138516    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 23:48:40.138567 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.138522    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 23:48:40.138567 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.138529    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 23:48:40.138567 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.138534    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 23:48:40.139213 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.139167    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 23:48:40.139213 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.139178    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 23:48:40.139213 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.139188    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 23:48:40.139213 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.139196    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 23:48:40.139213 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.139208    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 23:48:40.139413 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.139220    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 23:48:40.140173 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.140161    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 23:48:40.140213 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.140182    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 23:48:40.144019 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.143995    2556 server.go:1268] "Started kubelet"
Sep 09 23:48:40.144194 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.144097    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 23:48:40.144244 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.144217    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 23:48:40.144668 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.144644    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 23:48:40.144914 ip-10-0-139-211 systemd[1]: Started Kubernetes Kubelet.
Sep 09 23:48:40.147902 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.147779    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 23:48:40.147999 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.147878    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-211.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 23:48:40.148840 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.148603    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 23:48:40.149158 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.149130    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 23:48:40.150684 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.150667    2556 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 23:48:40.151767 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.151729    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 23:48:40.153641 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.152347    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-211.ec2.internal.18d3cbf2ad2a0314  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-211.ec2.internal,UID:ip-10-0-139-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-211.ec2.internal,},FirstTimestamp:2026-09-09 23:48:40.143954708 +0000 UTC m=+0.395090742,LastTimestamp:2026-09-09 23:48:40.143954708 +0000 UTC m=+0.395090742,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-211.ec2.internal,}"
Sep 09 23:48:40.156240 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.156219    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 23:48:40.156911 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.156889    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 23:48:40.157068 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.157038    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:40.157656 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157632    2556 factory.go:55] Registering systemd factory
Sep 09 23:48:40.157656 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157649    2556 factory.go:223] Registration of the systemd container factory successfully
Sep 09 23:48:40.157769 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157684    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 23:48:40.157769 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157686    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 23:48:40.157769 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157707    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 23:48:40.157889 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157826    2556 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 23:48:40.157889 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157835    2556 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 23:48:40.157889 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157847    2556 factory.go:153] Registering CRI-O factory
Sep 09 23:48:40.157889 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157858    2556 factory.go:223] Registration of the crio container factory successfully
Sep 09 23:48:40.158006 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157894    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 23:48:40.158006 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157911    2556 factory.go:103] Registering Raw factory
Sep 09 23:48:40.158006 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.157922    2556 manager.go:1196] Started watching for new ooms in manager
Sep 09 23:48:40.158290 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.158278    2556 manager.go:349] Starting recovery of all containers
Sep 09 23:48:40.162827 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.162800    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 23:48:40.162919 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.162877    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 23:48:40.169622 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.169607    2556 manager.go:354] Recovery completed
Sep 09 23:48:40.171216 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.171146    2556 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 23:48:40.174389 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.174375    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:40.176154 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.176132    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:40.176228 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.176166    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:40.176228 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.176185    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:40.176752 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.176738    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 23:48:40.176752 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.176752    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 23:48:40.176871 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.176756    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:40.176871 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.176769    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 23:48:40.177520 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.177503    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:40.177597 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.177533    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:40.177597 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.177546    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:40.177725 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.177656    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-211.ec2.internal.18d3cbf2af1556a7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-211.ec2.internal,UID:ip-10-0-139-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-211.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-211.ec2.internal,},FirstTimestamp:2026-09-09 23:48:40.176154279 +0000 UTC m=+0.427290313,LastTimestamp:2026-09-09 23:48:40.176154279 +0000 UTC m=+0.427290313,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-211.ec2.internal,}"
Sep 09 23:48:40.178963 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.178949    2556 policy_none.go:49] "None policy: Start"
Sep 09 23:48:40.179018 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.178968    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 23:48:40.179018 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.178980    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 23:48:40.179864 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.179844    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kmjtb"
Sep 09 23:48:40.180962 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.180951    2556 policy_none.go:47] "Start"
Sep 09 23:48:40.186877 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.186797    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-211.ec2.internal.18d3cbf2af1597ab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-211.ec2.internal,UID:ip-10-0-139-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-211.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-211.ec2.internal,},FirstTimestamp:2026-09-09 23:48:40.176170923 +0000 UTC m=+0.427306958,LastTimestamp:2026-09-09 23:48:40.176170923 +0000 UTC m=+0.427306958,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-211.ec2.internal,}"
Sep 09 23:48:40.187907 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.187888    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kmjtb"
Sep 09 23:48:40.210745 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.210728    2556 manager.go:337] "Starting Device Plugin manager"
Sep 09 23:48:40.237400 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.210776    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 23:48:40.237400 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.210788    2556 server.go:87] "Starting device plugin registration server"
Sep 09 23:48:40.237400 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.211007    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 23:48:40.237400 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.211017    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 23:48:40.237400 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.211240    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 23:48:40.237400 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.211312    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 23:48:40.237400 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.211320    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 23:48:40.237400 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.211809    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 23:48:40.237400 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.211859    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:40.260212 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.260187    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 23:48:40.261354 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.261332    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 23:48:40.261446 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.261376    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 23:48:40.261446 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.261397    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 23:48:40.261544 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.261433    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 23:48:40.264096 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.264077    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 23:48:40.311141 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.311098    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:40.312258 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.312181    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:40.312258 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.312216    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:40.312258 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.312228    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:40.312258 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.312252    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.321183 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.321168    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.321241 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.321191    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-211.ec2.internal\": node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:40.352777 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.352754    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:40.362085 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.362047    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-211.ec2.internal"]
Sep 09 23:48:40.362157 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.362128    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:40.363017 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.362999    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:40.363119 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.363033    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:40.363119 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.363047    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:40.365586 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.365572    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:40.365729 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.365715    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.365801 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.365750    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:40.366323 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.366311    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:40.366390 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.366327    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:40.366390 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.366374    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:40.366390 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.366385    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:40.366502 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.366333    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:40.366502 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.366411    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:40.368522 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.368507    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.368569 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.368532    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:40.369324 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.369307    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:40.369392 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.369338    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:40.369392 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.369355    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:40.399664 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.399642    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-211.ec2.internal\" not found" node="ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.404024 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.404009    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-211.ec2.internal\" not found" node="ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.453589 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.453567    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:40.459758 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.459704    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e4d623e27107cc13e36b7af8ad736df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal\" (UID: \"2e4d623e27107cc13e36b7af8ad736df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.459837 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.459765    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e4d623e27107cc13e36b7af8ad736df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal\" (UID: \"2e4d623e27107cc13e36b7af8ad736df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.459837 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.459805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44b8b5997f7574778e566a518d72e361-config\") pod \"kube-apiserver-proxy-ip-10-0-139-211.ec2.internal\" (UID: \"44b8b5997f7574778e566a518d72e361\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.553873 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.553844    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:40.560214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.560198    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e4d623e27107cc13e36b7af8ad736df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal\" (UID: \"2e4d623e27107cc13e36b7af8ad736df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.560268 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.560231    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44b8b5997f7574778e566a518d72e361-config\") pod \"kube-apiserver-proxy-ip-10-0-139-211.ec2.internal\" (UID: \"44b8b5997f7574778e566a518d72e361\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.560268 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.560243    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e4d623e27107cc13e36b7af8ad736df-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal\" (UID: \"2e4d623e27107cc13e36b7af8ad736df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.560268 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.560250    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e4d623e27107cc13e36b7af8ad736df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal\" (UID: \"2e4d623e27107cc13e36b7af8ad736df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.560360 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.560283    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e4d623e27107cc13e36b7af8ad736df-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal\" (UID: \"2e4d623e27107cc13e36b7af8ad736df\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.560360 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.560301    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44b8b5997f7574778e566a518d72e361-config\") pod \"kube-apiserver-proxy-ip-10-0-139-211.ec2.internal\" (UID: \"44b8b5997f7574778e566a518d72e361\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.654648 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.654576    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:40.702044 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.702022    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.707108 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:40.706937    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:40.755785 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.755634    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:40.856884 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.856847    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:40.964558 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:40.964530    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:41.065706 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:41.065662    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:41.087479 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.087451    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 23:48:41.087614 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.087596    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 23:48:41.166647 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:41.166616    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-211.ec2.internal\" not found"
Sep 09 23:48:41.190870 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.190830    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 23:43:41 +0000 UTC" deadline="2028-02-16 08:23:08.560138237 +0000 UTC"
Sep 09 23:48:41.190870 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.190862    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12584h34m27.369279808s"
Sep 09 23:48:41.194109 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.194092    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 23:48:41.195370 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.195351    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 23:48:41.195456 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.195386    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 23:48:41.195456 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.195402    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeHasSufficientPID"
Sep 09 23:48:41.201466 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.201444    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 23:48:41.225756 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.225696    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 23:48:41.257953 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.257923    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:41.271855 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.271838    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 23:48:41.272876 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.272862    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-211.ec2.internal"
Sep 09 23:48:41.280227 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.280212    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 23:48:41.310720 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:41.310691    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4d623e27107cc13e36b7af8ad736df.slice/crio-4691d5c9990562b8f11fcd498126f964587ea43f921123afd3df22ba9dc0d335 WatchSource:0}: Error finding container 4691d5c9990562b8f11fcd498126f964587ea43f921123afd3df22ba9dc0d335: Status 404 returned error can't find the container with id 4691d5c9990562b8f11fcd498126f964587ea43f921123afd3df22ba9dc0d335
Sep 09 23:48:41.315030 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.315014    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 23:48:41.677202 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:41.677128    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 23:48:42.133563 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.133530    2556 apiserver.go:52] "Watching apiserver"
Sep 09 23:48:42.140650 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.140623    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 23:48:42.141247 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.141203    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-211.ec2.internal","openshift-dns/node-resolver-wj527","openshift-image-registry/node-ca-l72vc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal","openshift-multus/multus-additional-cni-plugins-dr7db","openshift-network-operator/iptables-alerter-6k6xm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t","openshift-cluster-node-tuning-operator/tuned-l7btc","openshift-multus/multus-2w8q5","openshift-multus/network-metrics-daemon-plj9c","openshift-network-diagnostics/network-check-target-bgd5b","openshift-ovn-kubernetes/ovnkube-node-k9kxr","kube-system/global-pull-secret-syncer-kg65s","kube-system/konnectivity-agent-7nmdr"]
Sep 09 23:48:42.144724 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.144218    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:42.144724 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.144342    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:48:42.146845 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.146820    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:42.146982 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.146897    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:48:42.149837 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.149127    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l72vc"
Sep 09 23:48:42.151470 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.151329    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bxd9l\""
Sep 09 23:48:42.151904 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.151880    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 23:48:42.152120 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.152102    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 23:48:42.152343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.152325    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 23:48:42.152558 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.152542    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:48:42.154888 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.154865    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 23:48:42.155017 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.154956    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.155298 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.155275    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c8lr4\""
Sep 09 23:48:42.155729 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.155552    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 23:48:42.156914 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.156894    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6k6xm"
Sep 09 23:48:42.157050 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.157034    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 23:48:42.158411 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.158390    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 23:48:42.158518 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.158422    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-276v9\""
Sep 09 23:48:42.158587 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.158551    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 23:48:42.158845 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.158824    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 23:48:42.159161 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.159142    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 23:48:42.159600 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.159414    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 23:48:42.160268 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.160244    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.160382 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.160247    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 23:48:42.160950 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.160932    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 23:48:42.161279 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.161257    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 23:48:42.161711 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.161691    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kjppv\""
Sep 09 23:48:42.163643 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.163621    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 23:48:42.163859 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.163843    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.164044 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.163850    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kxbph\""
Sep 09 23:48:42.166165 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.166121    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7drwl\""
Sep 09 23:48:42.166359 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.166341    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 23:48:42.166433 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.166139    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wj527"
Sep 09 23:48:42.166497 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.166472    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 23:48:42.166660 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.166640    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 23:48:42.169470 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.169450    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 23:48:42.170512 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.170222    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 23:48:42.170768 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.170749    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p7ksc\""
Sep 09 23:48:42.172665 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.172644    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 23:48:42.173490 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.173472    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.174637 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.174614    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6cfm\" (UniqueName: \"kubernetes.io/projected/b62491be-c176-4206-9d7b-84caeae4c67d-kube-api-access-j6cfm\") pod \"node-ca-l72vc\" (UID: \"b62491be-c176-4206-9d7b-84caeae4c67d\") " pod="openshift-image-registry/node-ca-l72vc"
Sep 09 23:48:42.174731 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.174664    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d89fc94d-6596-40b6-8565-5bf0277d14da-agent-certs\") pod \"konnectivity-agent-7nmdr\" (UID: \"d89fc94d-6596-40b6-8565-5bf0277d14da\") " pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:48:42.174731 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.174702    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d89fc94d-6596-40b6-8565-5bf0277d14da-konnectivity-ca\") pod \"konnectivity-agent-7nmdr\" (UID: \"d89fc94d-6596-40b6-8565-5bf0277d14da\") " pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:48:42.179242 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179220    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:42.179338 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.179310    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:48:42.179615 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179593    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-system-cni-dir\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.179694 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179642    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.179694 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179670    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-cni-dir\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.179794 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179705    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-cnibin\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.179794 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179739    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-etc-kubernetes\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.179794 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179774    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab3c838b-f89a-4dd8-8318-705775e5bb2a-kubelet-config\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:42.179934 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179809    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3cd21a19-cfa8-463f-87e8-b1fe25b0c247-iptables-alerter-script\") pod \"iptables-alerter-6k6xm\" (UID: \"3cd21a19-cfa8-463f-87e8-b1fe25b0c247\") " pod="openshift-network-operator/iptables-alerter-6k6xm"
Sep 09 23:48:42.179934 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179842    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cd21a19-cfa8-463f-87e8-b1fe25b0c247-host-slash\") pod \"iptables-alerter-6k6xm\" (UID: \"3cd21a19-cfa8-463f-87e8-b1fe25b0c247\") " pod="openshift-network-operator/iptables-alerter-6k6xm"
Sep 09 23:48:42.179934 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179869    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:42.179934 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179909    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8drz8\" (UniqueName: \"kubernetes.io/projected/21369195-b9f6-4653-ae29-d2a7a07eb1e2-kube-api-access-8drz8\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:42.180262 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.179951    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-run-multus-certs\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.180327 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.180295    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:42.180380 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.180326    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-cnibin\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.180572 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.180298    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 23:48:42.181006 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.180984    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 23:48:42.181195 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.180991    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-87jk9\""
Sep 09 23:48:42.181617 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181593    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.181704 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181643    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-socket-dir-parent\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.181704 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181679    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-var-lib-kubelet\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.181813 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181713    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-hostroot\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.181813 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181747    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-conf-dir\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.181813 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b62491be-c176-4206-9d7b-84caeae4c67d-host\") pod \"node-ca-l72vc\" (UID: \"b62491be-c176-4206-9d7b-84caeae4c67d\") " pod="openshift-image-registry/node-ca-l72vc"
Sep 09 23:48:42.181813 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181808    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89h2f\" (UniqueName: \"kubernetes.io/projected/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-kube-api-access-89h2f\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.182033 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181842    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b62491be-c176-4206-9d7b-84caeae4c67d-serviceca\") pod \"node-ca-l72vc\" (UID: \"b62491be-c176-4206-9d7b-84caeae4c67d\") " pod="openshift-image-registry/node-ca-l72vc"
Sep 09 23:48:42.182033 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181874    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-cni-binary-copy\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.182033 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181901    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.182033 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181932    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c95v\" (UniqueName: \"kubernetes.io/projected/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-kube-api-access-7c95v\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.182033 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.181966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-system-cni-dir\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.182350 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.182324    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-run-k8s-cni-cncf-io\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.183013 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.182984    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-daemon-config\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.183166 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.183037    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.183166 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.183038    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-run-netns\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.183303 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.183175    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-var-lib-cni-multus\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.183303 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.183210    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab3c838b-f89a-4dd8-8318-705775e5bb2a-dbus\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:42.183303 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.183242    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfs49\" (UniqueName: \"kubernetes.io/projected/3cd21a19-cfa8-463f-87e8-b1fe25b0c247-kube-api-access-hfs49\") pod \"iptables-alerter-6k6xm\" (UID: \"3cd21a19-cfa8-463f-87e8-b1fe25b0c247\") " pod="openshift-network-operator/iptables-alerter-6k6xm"
Sep 09 23:48:42.183303 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.183280    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-cni-binary-copy\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.183516 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.183306    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-var-lib-cni-bin\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.185839 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.185709    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 23:48:42.185979 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.185960    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 23:48:42.186224 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.186203    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 23:48:42.186318 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.186240    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 23:48:42.186318 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.186211    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nm8xx\""
Sep 09 23:48:42.186749 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.186728    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 23:48:42.186869 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.186771    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 23:48:42.192631 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.192586    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tz2mx"
Sep 09 23:48:42.200594 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.200226    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tz2mx"
Sep 09 23:48:42.259417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.259397    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 23:48:42.265535 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.265497    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal" event={"ID":"2e4d623e27107cc13e36b7af8ad736df","Type":"ContainerStarted","Data":"4691d5c9990562b8f11fcd498126f964587ea43f921123afd3df22ba9dc0d335"}
Sep 09 23:48:42.266552 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.266531    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-211.ec2.internal" event={"ID":"44b8b5997f7574778e566a518d72e361","Type":"ContainerStarted","Data":"3e8bf35428f14709866cbf2843640993448536490a0bb60388af76e46a16ba0d"}
Sep 09 23:48:42.284363 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284330    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-tuned\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.284480 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284368    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-sys-fs\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.284480 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284397    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-run-openvswitch\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.284480 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284420    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-node-log\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.284480 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284464    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f387b325-e109-4ea3-af39-59292992adb5-ovn-node-metrics-cert\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.284709 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-modprobe-d\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.284709 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284545    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-var-lib-cni-bin\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.284709 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284574    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-systemd-units\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.284709 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284591    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-etc-openvswitch\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.284709 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284639    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6cfm\" (UniqueName: \"kubernetes.io/projected/b62491be-c176-4206-9d7b-84caeae4c67d-kube-api-access-j6cfm\") pod \"node-ca-l72vc\" (UID: \"b62491be-c176-4206-9d7b-84caeae4c67d\") " pod="openshift-image-registry/node-ca-l72vc"
Sep 09 23:48:42.284709 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284650    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-var-lib-cni-bin\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.284709 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284667    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-system-cni-dir\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.284709 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284688    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284727    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-cnibin\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284747    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-system-cni-dir\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284799    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284827    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-sysconfig\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.284846    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284850    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-tmp\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284882    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.284923    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-cnibin\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.284964    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.285121    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs podName:21369195-b9f6-4653-ae29-d2a7a07eb1e2 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:42.784893006 +0000 UTC m=+3.036029030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs") pod "network-metrics-daemon-plj9c" (UID: "21369195-b9f6-4653-ae29-d2a7a07eb1e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:42.285138 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285139    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-run-netns\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285161    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-log-socket\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285182    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-cnibin\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285212    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.285240    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret podName:ab3c838b-f89a-4dd8-8318-705775e5bb2a nodeName:}" failed. No retries permitted until 2026-09-09 23:48:42.785218914 +0000 UTC m=+3.036354952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret") pod "global-pull-secret-syncer-kg65s" (UID: "ab3c838b-f89a-4dd8-8318-705775e5bb2a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285276    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5799q\" (UniqueName: \"kubernetes.io/projected/555a4739-f6bd-4762-9564-7593b7a4f04e-kube-api-access-5799q\") pod \"node-resolver-wj527\" (UID: \"555a4739-f6bd-4762-9564-7593b7a4f04e\") " pod="openshift-dns/node-resolver-wj527"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285304    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-kubelet\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285330    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-var-lib-openvswitch\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285354    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285413    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt9n6\" (UniqueName: \"kubernetes.io/projected/f387b325-e109-4ea3-af39-59292992adb5-kube-api-access-pt9n6\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285446    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-cnibin\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285457    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffg44\" (UniqueName: \"kubernetes.io/projected/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-kube-api-access-ffg44\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285494    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b62491be-c176-4206-9d7b-84caeae4c67d-host\") pod \"node-ca-l72vc\" (UID: \"b62491be-c176-4206-9d7b-84caeae4c67d\") " pod="openshift-image-registry/node-ca-l72vc"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285530    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b62491be-c176-4206-9d7b-84caeae4c67d-host\") pod \"node-ca-l72vc\" (UID: \"b62491be-c176-4206-9d7b-84caeae4c67d\") " pod="openshift-image-registry/node-ca-l72vc"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285562    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-slash\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285587    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f387b325-e109-4ea3-af39-59292992adb5-ovnkube-script-lib\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.285841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285613    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-kubernetes\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285637    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-systemd\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285662    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285675    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285689    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b62491be-c176-4206-9d7b-84caeae4c67d-serviceca\") pod \"node-ca-l72vc\" (UID: \"b62491be-c176-4206-9d7b-84caeae4c67d\") " pod="openshift-image-registry/node-ca-l72vc"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285731    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-cni-binary-copy\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285759    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285785    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c95v\" (UniqueName: \"kubernetes.io/projected/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-kube-api-access-7c95v\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285811    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-run-k8s-cni-cncf-io\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285836    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-daemon-config\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285865    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f387b325-e109-4ea3-af39-59292992adb5-env-overrides\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285909    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-run-netns\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285935    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-var-lib-cni-multus\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285958    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285960    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfs49\" (UniqueName: \"kubernetes.io/projected/3cd21a19-cfa8-463f-87e8-b1fe25b0c247-kube-api-access-hfs49\") pod \"iptables-alerter-6k6xm\" (UID: \"3cd21a19-cfa8-463f-87e8-b1fe25b0c247\") " pod="openshift-network-operator/iptables-alerter-6k6xm"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285979    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.286514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.285994    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555a4739-f6bd-4762-9564-7593b7a4f04e-tmp-dir\") pod \"node-resolver-wj527\" (UID: \"555a4739-f6bd-4762-9564-7593b7a4f04e\") " pod="openshift-dns/node-resolver-wj527"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286019    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lf7x\" (UniqueName: \"kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x\") pod \"network-check-target-bgd5b\" (UID: \"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab\") " pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286054    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-cni-bin\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286071    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-sysctl-d\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286085    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-device-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286104    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-cni-binary-copy\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286121    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d89fc94d-6596-40b6-8565-5bf0277d14da-agent-certs\") pod \"konnectivity-agent-7nmdr\" (UID: \"d89fc94d-6596-40b6-8565-5bf0277d14da\") " pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286150    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d89fc94d-6596-40b6-8565-5bf0277d14da-konnectivity-ca\") pod \"konnectivity-agent-7nmdr\" (UID: \"d89fc94d-6596-40b6-8565-5bf0277d14da\") " pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286166    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-cni-dir\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286202    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab3c838b-f89a-4dd8-8318-705775e5bb2a-kubelet-config\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286219    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3cd21a19-cfa8-463f-87e8-b1fe25b0c247-iptables-alerter-script\") pod \"iptables-alerter-6k6xm\" (UID: \"3cd21a19-cfa8-463f-87e8-b1fe25b0c247\") " pod="openshift-network-operator/iptables-alerter-6k6xm"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286235    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cd21a19-cfa8-463f-87e8-b1fe25b0c247-host-slash\") pod \"iptables-alerter-6k6xm\" (UID: \"3cd21a19-cfa8-463f-87e8-b1fe25b0c247\") " pod="openshift-network-operator/iptables-alerter-6k6xm"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286251    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8drz8\" (UniqueName: \"kubernetes.io/projected/21369195-b9f6-4653-ae29-d2a7a07eb1e2-kube-api-access-8drz8\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286270    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-etc-kubernetes\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286270    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b62491be-c176-4206-9d7b-84caeae4c67d-serviceca\") pod \"node-ca-l72vc\" (UID: \"b62491be-c176-4206-9d7b-84caeae4c67d\") " pod="openshift-image-registry/node-ca-l72vc"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286286    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/555a4739-f6bd-4762-9564-7593b7a4f04e-hosts-file\") pod \"node-resolver-wj527\" (UID: \"555a4739-f6bd-4762-9564-7593b7a4f04e\") " pod="openshift-dns/node-resolver-wj527"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286302    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-host\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.287371 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286316    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-run-k8s-cni-cncf-io\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286317    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-socket-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286367    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-registration-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286398    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-etc-selinux\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286412    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-var-lib-cni-multus\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286433    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-run-multus-certs\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286463    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-var-lib-kubelet\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286479    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-cni-binary-copy\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286492    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-socket-dir-parent\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286518    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-var-lib-kubelet\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286544    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-hostroot\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-run\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286598    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7s6m\" (UniqueName: \"kubernetes.io/projected/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-kube-api-access-t7s6m\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286369    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-run-netns\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286642    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-conf-dir\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286669    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89h2f\" (UniqueName: \"kubernetes.io/projected/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-kube-api-access-89h2f\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286693    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-run-ovn\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.288325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286718    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-cni-netd\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286745    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-cni-binary-copy\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286746    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f387b325-e109-4ea3-af39-59292992adb5-ovnkube-config\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286787    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-sysctl-conf\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286812    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-sys\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-lib-modules\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286872    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-system-cni-dir\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286899    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab3c838b-f89a-4dd8-8318-705775e5bb2a-dbus\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286924    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-run-systemd\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286982    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3cd21a19-cfa8-463f-87e8-b1fe25b0c247-iptables-alerter-script\") pod \"iptables-alerter-6k6xm\" (UID: \"3cd21a19-cfa8-463f-87e8-b1fe25b0c247\") " pod="openshift-network-operator/iptables-alerter-6k6xm"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.286994    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-daemon-config\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287065    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-run-multus-certs\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287103    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3cd21a19-cfa8-463f-87e8-b1fe25b0c247-host-slash\") pod \"iptables-alerter-6k6xm\" (UID: \"3cd21a19-cfa8-463f-87e8-b1fe25b0c247\") " pod="openshift-network-operator/iptables-alerter-6k6xm"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287211    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-hostroot\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287147    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-socket-dir-parent\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287168    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab3c838b-f89a-4dd8-8318-705775e5bb2a-kubelet-config\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287178    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-host-var-lib-kubelet\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287190    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d89fc94d-6596-40b6-8565-5bf0277d14da-konnectivity-ca\") pod \"konnectivity-agent-7nmdr\" (UID: \"d89fc94d-6596-40b6-8565-5bf0277d14da\") " pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:48:42.289214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287233    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-cni-dir\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.290216 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287280    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab3c838b-f89a-4dd8-8318-705775e5bb2a-dbus\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:42.290216 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287260    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-system-cni-dir\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.290216 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287133    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-etc-kubernetes\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.290216 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.287319    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-multus-conf-dir\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.290216 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.288218    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 23:48:42.293171 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.293126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d89fc94d-6596-40b6-8565-5bf0277d14da-agent-certs\") pod \"konnectivity-agent-7nmdr\" (UID: \"d89fc94d-6596-40b6-8565-5bf0277d14da\") " pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:48:42.294981 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.294921    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c95v\" (UniqueName: \"kubernetes.io/projected/033a3818-5580-4cc4-9e17-a1d5fd0d6a71-kube-api-access-7c95v\") pod \"multus-additional-cni-plugins-dr7db\" (UID: \"033a3818-5580-4cc4-9e17-a1d5fd0d6a71\") " pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.295376 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.295323    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6cfm\" (UniqueName: \"kubernetes.io/projected/b62491be-c176-4206-9d7b-84caeae4c67d-kube-api-access-j6cfm\") pod \"node-ca-l72vc\" (UID: \"b62491be-c176-4206-9d7b-84caeae4c67d\") " pod="openshift-image-registry/node-ca-l72vc"
Sep 09 23:48:42.297248 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.296690    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfs49\" (UniqueName: \"kubernetes.io/projected/3cd21a19-cfa8-463f-87e8-b1fe25b0c247-kube-api-access-hfs49\") pod \"iptables-alerter-6k6xm\" (UID: \"3cd21a19-cfa8-463f-87e8-b1fe25b0c247\") " pod="openshift-network-operator/iptables-alerter-6k6xm"
Sep 09 23:48:42.297783 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.297755    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8drz8\" (UniqueName: \"kubernetes.io/projected/21369195-b9f6-4653-ae29-d2a7a07eb1e2-kube-api-access-8drz8\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:42.298098 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.298076    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89h2f\" (UniqueName: \"kubernetes.io/projected/0c59eba6-c8ab-4a3b-8c89-dad9d27a3922-kube-api-access-89h2f\") pod \"multus-2w8q5\" (UID: \"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922\") " pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.387974 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.387885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f387b325-e109-4ea3-af39-59292992adb5-env-overrides\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.387974 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.387927    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555a4739-f6bd-4762-9564-7593b7a4f04e-tmp-dir\") pod \"node-resolver-wj527\" (UID: \"555a4739-f6bd-4762-9564-7593b7a4f04e\") " pod="openshift-dns/node-resolver-wj527"
Sep 09 23:48:42.387974 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.387947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lf7x\" (UniqueName: \"kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x\") pod \"network-check-target-bgd5b\" (UID: \"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab\") " pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:42.388299 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-cni-bin\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.388299 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388225    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-sysctl-d\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.388299 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388254    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-device-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.388299 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388295    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/555a4739-f6bd-4762-9564-7593b7a4f04e-hosts-file\") pod \"node-resolver-wj527\" (UID: \"555a4739-f6bd-4762-9564-7593b7a4f04e\") " pod="openshift-dns/node-resolver-wj527"
Sep 09 23:48:42.388517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388310    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-cni-bin\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.388517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388325    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-host\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.388517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388335    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555a4739-f6bd-4762-9564-7593b7a4f04e-tmp-dir\") pod \"node-resolver-wj527\" (UID: \"555a4739-f6bd-4762-9564-7593b7a4f04e\") " pod="openshift-dns/node-resolver-wj527"
Sep 09 23:48:42.388517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388371    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-socket-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.388517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388401    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-registration-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.388517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388411    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-host\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.388517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-etc-selinux\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.388517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388448    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/555a4739-f6bd-4762-9564-7593b7a4f04e-hosts-file\") pod \"node-resolver-wj527\" (UID: \"555a4739-f6bd-4762-9564-7593b7a4f04e\") " pod="openshift-dns/node-resolver-wj527"
Sep 09 23:48:42.388517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388506    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-var-lib-kubelet\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.388517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388453    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-var-lib-kubelet\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388545    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-run\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388563    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7s6m\" (UniqueName: \"kubernetes.io/projected/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-kube-api-access-t7s6m\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388559    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f387b325-e109-4ea3-af39-59292992adb5-env-overrides\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388585    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-run-ovn\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388568    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-sysctl-d\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388598    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-registration-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388614    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-cni-netd\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388641    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f387b325-e109-4ea3-af39-59292992adb5-ovnkube-config\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388404    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-device-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388650    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-run-ovn\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388682    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-sysctl-conf\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388699    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-run\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388708    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-sys\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388708    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-socket-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388731    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-lib-modules\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388742    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-cni-netd\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388641    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-etc-selinux\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388758    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-run-systemd\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388779    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-tuned\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388799    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-sys-fs\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388805    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-sys\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-run-openvswitch\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388847    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-node-log\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388853    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-sysctl-conf\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388858    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-run-systemd\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388873    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f387b325-e109-4ea3-af39-59292992adb5-ovn-node-metrics-cert\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-lib-modules\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388912    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-sys-fs\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388913    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-run-openvswitch\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388938    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-node-log\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388948    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-modprobe-d\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.388982    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-systemd-units\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389009    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-etc-openvswitch\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.389945 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389054    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-sysconfig\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389108    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-tmp\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389160    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-run-netns\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389186    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-log-socket\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389195    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f387b325-e109-4ea3-af39-59292992adb5-ovnkube-config\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389216    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5799q\" (UniqueName: \"kubernetes.io/projected/555a4739-f6bd-4762-9564-7593b7a4f04e-kube-api-access-5799q\") pod \"node-resolver-wj527\" (UID: \"555a4739-f6bd-4762-9564-7593b7a4f04e\") " pod="openshift-dns/node-resolver-wj527"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389243    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-kubelet\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389273    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389292    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-run-netns\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389338    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-systemd-units\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389339    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-var-lib-openvswitch\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389294    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-var-lib-openvswitch\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389351    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-log-socket\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389354    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-sysconfig\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389408    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-kubelet\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389409    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-etc-openvswitch\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.390858 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389593    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-modprobe-d\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389625    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389644    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt9n6\" (UniqueName: \"kubernetes.io/projected/f387b325-e109-4ea3-af39-59292992adb5-kube-api-access-pt9n6\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389667    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffg44\" (UniqueName: \"kubernetes.io/projected/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-kube-api-access-ffg44\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389684    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-slash\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389690    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f387b325-e109-4ea3-af39-59292992adb5-host-slash\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f387b325-e109-4ea3-af39-59292992adb5-ovnkube-script-lib\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389802    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-kubernetes\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389826    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-systemd\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389851    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389920    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-kubernetes\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389964    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-systemd\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.389968    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.391736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.390454    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f387b325-e109-4ea3-af39-59292992adb5-ovnkube-script-lib\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.392451 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.392060    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-etc-tuned\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.392451 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.392155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f387b325-e109-4ea3-af39-59292992adb5-ovn-node-metrics-cert\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.392451 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.392374    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-tmp\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.394989 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.394967    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:48:42.394989 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.394992    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:48:42.395216 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.395005    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7lf7x for pod openshift-network-diagnostics/network-check-target-bgd5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:42.395216 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.395085    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x podName:7f5d6c3d-1bac-4d49-9b92-7f64c87939ab nodeName:}" failed. No retries permitted until 2026-09-09 23:48:42.895064097 +0000 UTC m=+3.146200144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7lf7x" (UniqueName: "kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x") pod "network-check-target-bgd5b" (UID: "7f5d6c3d-1bac-4d49-9b92-7f64c87939ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:42.397817 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.397794    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7s6m\" (UniqueName: \"kubernetes.io/projected/0c5d38a2-3207-4a3b-923c-faa6dcd7fae6-kube-api-access-t7s6m\") pod \"aws-ebs-csi-driver-node-wnj9t\" (UID: \"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.397952 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.397794    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5799q\" (UniqueName: \"kubernetes.io/projected/555a4739-f6bd-4762-9564-7593b7a4f04e-kube-api-access-5799q\") pod \"node-resolver-wj527\" (UID: \"555a4739-f6bd-4762-9564-7593b7a4f04e\") " pod="openshift-dns/node-resolver-wj527"
Sep 09 23:48:42.398670 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.398647    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffg44\" (UniqueName: \"kubernetes.io/projected/4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a-kube-api-access-ffg44\") pod \"tuned-l7btc\" (UID: \"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a\") " pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.399811 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.399780    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt9n6\" (UniqueName: \"kubernetes.io/projected/f387b325-e109-4ea3-af39-59292992adb5-kube-api-access-pt9n6\") pod \"ovnkube-node-k9kxr\" (UID: \"f387b325-e109-4ea3-af39-59292992adb5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.464952 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.464925    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l72vc"
Sep 09 23:48:42.479973 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.479948    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:48:42.492160 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.492121    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dr7db"
Sep 09 23:48:42.499923 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.499903    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6k6xm"
Sep 09 23:48:42.510738 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.510721    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2w8q5"
Sep 09 23:48:42.515630 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.515613    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 23:48:42.518690 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.518673    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t"
Sep 09 23:48:42.528532 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.528515    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wj527"
Sep 09 23:48:42.536502 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.536484    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l7btc"
Sep 09 23:48:42.544464 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.544447    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:48:42.792274 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.792240    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:42.792475 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.792286    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:42.792475 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.792411    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:42.792475 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.792452    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:42.792644 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.792487    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs podName:21369195-b9f6-4653-ae29-d2a7a07eb1e2 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:43.792465471 +0000 UTC m=+4.043601518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs") pod "network-metrics-daemon-plj9c" (UID: "21369195-b9f6-4653-ae29-d2a7a07eb1e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:42.792644 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.792507    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret podName:ab3c838b-f89a-4dd8-8318-705775e5bb2a nodeName:}" failed. No retries permitted until 2026-09-09 23:48:43.792497326 +0000 UTC m=+4.043633348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret") pod "global-pull-secret-syncer-kg65s" (UID: "ab3c838b-f89a-4dd8-8318-705775e5bb2a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:42.993337 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:42.993301    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lf7x\" (UniqueName: \"kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x\") pod \"network-check-target-bgd5b\" (UID: \"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab\") " pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:42.993506 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.993434    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:48:42.993506 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.993449    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:48:42.993506 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.993459    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7lf7x for pod openshift-network-diagnostics/network-check-target-bgd5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:42.993627 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:42.993513    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x podName:7f5d6c3d-1bac-4d49-9b92-7f64c87939ab nodeName:}" failed. No retries permitted until 2026-09-09 23:48:43.99349688 +0000 UTC m=+4.244632902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lf7x" (UniqueName: "kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x") pod "network-check-target-bgd5b" (UID: "7f5d6c3d-1bac-4d49-9b92-7f64c87939ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:43.203685 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:43.203642    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:43:42 +0000 UTC" deadline="2028-02-07 14:33:40.770712088 +0000 UTC"
Sep 09 23:48:43.203685 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:43.203672    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12374h44m57.567044799s"
Sep 09 23:48:43.262387 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:43.262350    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:43.262555 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:43.262350    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:43.262555 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:43.262511    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:48:43.262555 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:43.262550    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:48:43.383820 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:43.383786    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5d38a2_3207_4a3b_923c_faa6dcd7fae6.slice/crio-fcdc020368e10db88cf0515b167092a8848d0e858880e5f3499c9922705f540d WatchSource:0}: Error finding container fcdc020368e10db88cf0515b167092a8848d0e858880e5f3499c9922705f540d: Status 404 returned error can't find the container with id fcdc020368e10db88cf0515b167092a8848d0e858880e5f3499c9922705f540d
Sep 09 23:48:43.385747 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:43.385540    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c59eba6_c8ab_4a3b_8c89_dad9d27a3922.slice/crio-7085d6d4594c7679b55edf275c25a727a652291a2864daec2bafe40a05791f98 WatchSource:0}: Error finding container 7085d6d4594c7679b55edf275c25a727a652291a2864daec2bafe40a05791f98: Status 404 returned error can't find the container with id 7085d6d4594c7679b55edf275c25a727a652291a2864daec2bafe40a05791f98
Sep 09 23:48:43.389291 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:43.389248    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf387b325_e109_4ea3_af39_59292992adb5.slice/crio-3e8b1e706b42075f2f6a53285f37d71c60931d67b28c231004d698847b739f30 WatchSource:0}: Error finding container 3e8b1e706b42075f2f6a53285f37d71c60931d67b28c231004d698847b739f30: Status 404 returned error can't find the container with id 3e8b1e706b42075f2f6a53285f37d71c60931d67b28c231004d698847b739f30
Sep 09 23:48:43.390173 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:43.390142    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efb074d_6bb6_412a_8cfe_f70fcaaa1c5a.slice/crio-3dd6db07976153554db464292466484358e1de6e34a73e2ae7df2721a98159df WatchSource:0}: Error finding container 3dd6db07976153554db464292466484358e1de6e34a73e2ae7df2721a98159df: Status 404 returned error can't find the container with id 3dd6db07976153554db464292466484358e1de6e34a73e2ae7df2721a98159df
Sep 09 23:48:43.391318 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:43.391299    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555a4739_f6bd_4762_9564_7593b7a4f04e.slice/crio-761773a4685df4dae706083c7f6e53300cdda2b736e6f132c03cbc286f8bfe8e WatchSource:0}: Error finding container 761773a4685df4dae706083c7f6e53300cdda2b736e6f132c03cbc286f8bfe8e: Status 404 returned error can't find the container with id 761773a4685df4dae706083c7f6e53300cdda2b736e6f132c03cbc286f8bfe8e
Sep 09 23:48:43.392167 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:43.392140    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62491be_c176_4206_9d7b_84caeae4c67d.slice/crio-676317ad7f5f2068d8eca8bce0a8a3168cf91f86324bc96a92a05bfdfd83fe03 WatchSource:0}: Error finding container 676317ad7f5f2068d8eca8bce0a8a3168cf91f86324bc96a92a05bfdfd83fe03: Status 404 returned error can't find the container with id 676317ad7f5f2068d8eca8bce0a8a3168cf91f86324bc96a92a05bfdfd83fe03
Sep 09 23:48:43.393291 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:43.393170    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89fc94d_6596_40b6_8565_5bf0277d14da.slice/crio-fe897d94212a5e420ba0012ece80de00836cb70c2731811551354df3653badfc WatchSource:0}: Error finding container fe897d94212a5e420ba0012ece80de00836cb70c2731811551354df3653badfc: Status 404 returned error can't find the container with id fe897d94212a5e420ba0012ece80de00836cb70c2731811551354df3653badfc
Sep 09 23:48:43.395451 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:43.395282    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd21a19_cfa8_463f_87e8_b1fe25b0c247.slice/crio-0941d476fee3e49c6c4871dd4243aab8ef9b896dc1192ce6a56879a24af37b86 WatchSource:0}: Error finding container 0941d476fee3e49c6c4871dd4243aab8ef9b896dc1192ce6a56879a24af37b86: Status 404 returned error can't find the container with id 0941d476fee3e49c6c4871dd4243aab8ef9b896dc1192ce6a56879a24af37b86
Sep 09 23:48:43.397783 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:48:43.396358    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033a3818_5580_4cc4_9e17_a1d5fd0d6a71.slice/crio-b7bcd895e76c6dd86e03c5001210a8690d658179d239026ac50f3e245ed7a5d9 WatchSource:0}: Error finding container b7bcd895e76c6dd86e03c5001210a8690d658179d239026ac50f3e245ed7a5d9: Status 404 returned error can't find the container with id b7bcd895e76c6dd86e03c5001210a8690d658179d239026ac50f3e245ed7a5d9
Sep 09 23:48:43.798794 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:43.798554    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:43.798942 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:43.798817    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:43.798942 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:43.798732    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:43.798942 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:43.798916    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs podName:21369195-b9f6-4653-ae29-d2a7a07eb1e2 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:45.798891729 +0000 UTC m=+6.050027775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs") pod "network-metrics-daemon-plj9c" (UID: "21369195-b9f6-4653-ae29-d2a7a07eb1e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:43.799121 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:43.798969    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:43.799121 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:43.799018    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret podName:ab3c838b-f89a-4dd8-8318-705775e5bb2a nodeName:}" failed. No retries permitted until 2026-09-09 23:48:45.799006502 +0000 UTC m=+6.050142539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret") pod "global-pull-secret-syncer-kg65s" (UID: "ab3c838b-f89a-4dd8-8318-705775e5bb2a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:44.001699 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.001623    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lf7x\" (UniqueName: \"kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x\") pod \"network-check-target-bgd5b\" (UID: \"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab\") " pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:44.001906 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:44.001851    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:48:44.001906 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:44.001879    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:48:44.001906 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:44.001894    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7lf7x for pod openshift-network-diagnostics/network-check-target-bgd5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:44.002097 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:44.001958    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x podName:7f5d6c3d-1bac-4d49-9b92-7f64c87939ab nodeName:}" failed. No retries permitted until 2026-09-09 23:48:46.001937263 +0000 UTC m=+6.253073302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lf7x" (UniqueName: "kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x") pod "network-check-target-bgd5b" (UID: "7f5d6c3d-1bac-4d49-9b92-7f64c87939ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:44.204300 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.204198    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:43:42 +0000 UTC" deadline="2028-06-15 12:10:45.5313219 +0000 UTC"
Sep 09 23:48:44.204300 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.204266    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15468h22m1.327060358s"
Sep 09 23:48:44.263049 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.262410    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:44.263049 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:44.262540    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:48:44.286714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.285816    2556 generic.go:356] "Generic (PLEG): container finished" podID="2e4d623e27107cc13e36b7af8ad736df" containerID="b801c128e746e3f25ed47ff584406fba4baa8b61c6f7db3d54478909287fcee3" exitCode=0
Sep 09 23:48:44.286714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.285904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal" event={"ID":"2e4d623e27107cc13e36b7af8ad736df","Type":"ContainerDied","Data":"b801c128e746e3f25ed47ff584406fba4baa8b61c6f7db3d54478909287fcee3"}
Sep 09 23:48:44.298972 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.298939    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-211.ec2.internal" event={"ID":"44b8b5997f7574778e566a518d72e361","Type":"ContainerStarted","Data":"265a969a6aac499d835eda0d24724a80c486e303c37eea188f4def91643618c0"}
Sep 09 23:48:44.316020 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.315846    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-211.ec2.internal" podStartSLOduration=2.315826365 podStartE2EDuration="2.315826365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:44.315313501 +0000 UTC m=+4.566449549" watchObservedRunningTime="2026-09-09 23:48:44.315826365 +0000 UTC m=+4.566962413"
Sep 09 23:48:44.317933 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.317802    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dr7db" event={"ID":"033a3818-5580-4cc4-9e17-a1d5fd0d6a71","Type":"ContainerStarted","Data":"b7bcd895e76c6dd86e03c5001210a8690d658179d239026ac50f3e245ed7a5d9"}
Sep 09 23:48:44.330532 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.330441    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6k6xm" event={"ID":"3cd21a19-cfa8-463f-87e8-b1fe25b0c247","Type":"ContainerStarted","Data":"0941d476fee3e49c6c4871dd4243aab8ef9b896dc1192ce6a56879a24af37b86"}
Sep 09 23:48:44.345559 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.345526    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l72vc" event={"ID":"b62491be-c176-4206-9d7b-84caeae4c67d","Type":"ContainerStarted","Data":"676317ad7f5f2068d8eca8bce0a8a3168cf91f86324bc96a92a05bfdfd83fe03"}
Sep 09 23:48:44.348552 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.348484    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2w8q5" event={"ID":"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922","Type":"ContainerStarted","Data":"7085d6d4594c7679b55edf275c25a727a652291a2864daec2bafe40a05791f98"}
Sep 09 23:48:44.352571 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.352492    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t" event={"ID":"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6","Type":"ContainerStarted","Data":"fcdc020368e10db88cf0515b167092a8848d0e858880e5f3499c9922705f540d"}
Sep 09 23:48:44.356753 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.356706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l7btc" event={"ID":"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a","Type":"ContainerStarted","Data":"3dd6db07976153554db464292466484358e1de6e34a73e2ae7df2721a98159df"}
Sep 09 23:48:44.368282 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.368189    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7nmdr" event={"ID":"d89fc94d-6596-40b6-8565-5bf0277d14da","Type":"ContainerStarted","Data":"fe897d94212a5e420ba0012ece80de00836cb70c2731811551354df3653badfc"}
Sep 09 23:48:44.375468 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.375406    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wj527" event={"ID":"555a4739-f6bd-4762-9564-7593b7a4f04e","Type":"ContainerStarted","Data":"761773a4685df4dae706083c7f6e53300cdda2b736e6f132c03cbc286f8bfe8e"}
Sep 09 23:48:44.388061 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:44.387967    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr" event={"ID":"f387b325-e109-4ea3-af39-59292992adb5","Type":"ContainerStarted","Data":"3e8b1e706b42075f2f6a53285f37d71c60931d67b28c231004d698847b739f30"}
Sep 09 23:48:45.262680 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:45.262598    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:45.263222 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:45.262760    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:48:45.263222 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:45.263187    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:45.263368 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:45.263288    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:48:45.405387 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:45.405288    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal" event={"ID":"2e4d623e27107cc13e36b7af8ad736df","Type":"ContainerStarted","Data":"3545015f7eff47dd173cb8389be089e818ca28932b5be0713bd06d54dee3d198"}
Sep 09 23:48:45.421302 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:45.421249    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-211.ec2.internal" podStartSLOduration=3.421227541 podStartE2EDuration="3.421227541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:48:45.420963315 +0000 UTC m=+5.672099364" watchObservedRunningTime="2026-09-09 23:48:45.421227541 +0000 UTC m=+5.672363587"
Sep 09 23:48:45.816492 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:45.816460    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:45.816683 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:45.816501    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:45.816683 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:45.816643    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:45.816806 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:45.816704    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret podName:ab3c838b-f89a-4dd8-8318-705775e5bb2a nodeName:}" failed. No retries permitted until 2026-09-09 23:48:49.816688267 +0000 UTC m=+10.067824293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret") pod "global-pull-secret-syncer-kg65s" (UID: "ab3c838b-f89a-4dd8-8318-705775e5bb2a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:45.817064 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:45.817015    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:45.817064 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:45.817061    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs podName:21369195-b9f6-4653-ae29-d2a7a07eb1e2 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:49.817050534 +0000 UTC m=+10.068186557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs") pod "network-metrics-daemon-plj9c" (UID: "21369195-b9f6-4653-ae29-d2a7a07eb1e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:46.018123 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:46.018018    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lf7x\" (UniqueName: \"kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x\") pod \"network-check-target-bgd5b\" (UID: \"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab\") " pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:46.018342 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:46.018218    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:48:46.018342 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:46.018240    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:48:46.018342 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:46.018254    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7lf7x for pod openshift-network-diagnostics/network-check-target-bgd5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:46.018507 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:46.018396    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x podName:7f5d6c3d-1bac-4d49-9b92-7f64c87939ab nodeName:}" failed. No retries permitted until 2026-09-09 23:48:50.018309906 +0000 UTC m=+10.269445949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lf7x" (UniqueName: "kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x") pod "network-check-target-bgd5b" (UID: "7f5d6c3d-1bac-4d49-9b92-7f64c87939ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:46.265151 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:46.265096    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:46.265637 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:46.265251    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:48:47.265827 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:47.265217    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:47.265827 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:47.265326    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:48:47.265827 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:47.265674    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:47.265827 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:47.265784    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:48:48.262594 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:48.262556    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:48.262803 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:48.262707    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:48:49.262599 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:49.262567    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:49.263063 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:49.262567    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:49.263063 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:49.262707    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:48:49.263063 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:49.262760    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:48:49.852777 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:49.852135    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:49.852777 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:49.852188    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:49.852777 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:49.852362    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:49.852777 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:49.852431    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret podName:ab3c838b-f89a-4dd8-8318-705775e5bb2a nodeName:}" failed. No retries permitted until 2026-09-09 23:48:57.852410183 +0000 UTC m=+18.103546228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret") pod "global-pull-secret-syncer-kg65s" (UID: "ab3c838b-f89a-4dd8-8318-705775e5bb2a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:49.852777 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:49.852522    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:49.852777 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:49.852574    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs podName:21369195-b9f6-4653-ae29-d2a7a07eb1e2 nodeName:}" failed. No retries permitted until 2026-09-09 23:48:57.85255982 +0000 UTC m=+18.103695858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs") pod "network-metrics-daemon-plj9c" (UID: "21369195-b9f6-4653-ae29-d2a7a07eb1e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:50.053922 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:50.053322    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lf7x\" (UniqueName: \"kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x\") pod \"network-check-target-bgd5b\" (UID: \"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab\") " pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:50.053922 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:50.053489    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:48:50.053922 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:50.053511    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:48:50.053922 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:50.053526    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7lf7x for pod openshift-network-diagnostics/network-check-target-bgd5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:50.053922 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:50.053589    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x podName:7f5d6c3d-1bac-4d49-9b92-7f64c87939ab nodeName:}" failed. No retries permitted until 2026-09-09 23:48:58.053568542 +0000 UTC m=+18.304704586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lf7x" (UniqueName: "kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x") pod "network-check-target-bgd5b" (UID: "7f5d6c3d-1bac-4d49-9b92-7f64c87939ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:50.263686 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:50.263470    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:50.263686 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:50.263580    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:48:51.262176 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:51.262136    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:51.262357 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:51.262299    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:48:51.262455 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:51.262436    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:51.262558 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:51.262539    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:48:52.262470 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:52.262165    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:52.262470 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:52.262319    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:48:53.261596 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:53.261561    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:53.261775 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:53.261561    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:53.261775 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:53.261682    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:48:53.261775 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:53.261755    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:48:54.262223 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:54.262198    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:54.262615 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:54.262329    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:48:55.262567 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:55.262533    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:55.262567 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:55.262565    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:55.263037 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:55.262655    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:48:55.263037 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:55.262785    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:48:56.261603 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:56.261577    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:56.261795 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:56.261689    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:48:57.262615 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:57.262575    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:57.263139 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:57.262632    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:57.263139 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:57.262720    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:48:57.263139 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:57.262839    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:48:57.912019 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:57.911934    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:57.912201 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:57.912095    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:57.912201 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:57.912102    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:57.912201 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:57.912170    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs podName:21369195-b9f6-4653-ae29-d2a7a07eb1e2 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:13.91214854 +0000 UTC m=+34.163284579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs") pod "network-metrics-daemon-plj9c" (UID: "21369195-b9f6-4653-ae29-d2a7a07eb1e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:48:57.912201 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:57.912181    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:57.912410 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:57.912235    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret podName:ab3c838b-f89a-4dd8-8318-705775e5bb2a nodeName:}" failed. No retries permitted until 2026-09-09 23:49:13.912219488 +0000 UTC m=+34.163355530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret") pod "global-pull-secret-syncer-kg65s" (UID: "ab3c838b-f89a-4dd8-8318-705775e5bb2a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:48:58.113521 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:58.113479    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lf7x\" (UniqueName: \"kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x\") pod \"network-check-target-bgd5b\" (UID: \"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab\") " pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:58.113683 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:58.113656    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:48:58.113683 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:58.113676    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:48:58.113767 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:58.113689    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7lf7x for pod openshift-network-diagnostics/network-check-target-bgd5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:58.113813 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:58.113767    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x podName:7f5d6c3d-1bac-4d49-9b92-7f64c87939ab nodeName:}" failed. No retries permitted until 2026-09-09 23:49:14.113750184 +0000 UTC m=+34.364886209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lf7x" (UniqueName: "kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x") pod "network-check-target-bgd5b" (UID: "7f5d6c3d-1bac-4d49-9b92-7f64c87939ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:48:58.262701 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:58.262656    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:48:58.263200 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:58.262788    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:48:59.262496 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:59.262456    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:48:59.262720 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:48:59.262516    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:48:59.262720 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:59.262606    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:48:59.263134 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:48:59.262746    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:49:00.262655 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:00.262625    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:00.262839 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:00.262724    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:49:01.262497 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:01.262461    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:49:01.262692 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:01.262505    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:01.262692 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:01.262625    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:49:01.262830 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:01.262773    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:49:02.262829 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.262478    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:02.263540 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:02.262949    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:49:02.446237 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.446202    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7nmdr" event={"ID":"d89fc94d-6596-40b6-8565-5bf0277d14da","Type":"ContainerStarted","Data":"91618ad3a1033a5e32bd2657051f30c4889e300d17c63e1945a2f5e19ccb599f"}
Sep 09 23:49:02.447848 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.447821    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wj527" event={"ID":"555a4739-f6bd-4762-9564-7593b7a4f04e","Type":"ContainerStarted","Data":"c79a7c1450dde1d6b7f9ad24275a723b387b6c9044d1a08054094f932cbd8f70"}
Sep 09 23:49:02.453930 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.453905    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr" event={"ID":"f387b325-e109-4ea3-af39-59292992adb5","Type":"ContainerStarted","Data":"0c59f1397be788c79ba93a36b9058e66dbff57fea3eaac492e6ee22cda73d90d"}
Sep 09 23:49:02.454073 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.453996    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr" event={"ID":"f387b325-e109-4ea3-af39-59292992adb5","Type":"ContainerStarted","Data":"1af0e924825977b1e0306ece011509027f87b5c5fb3343c4895c3740218ef8fd"}
Sep 09 23:49:02.454073 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.454046    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr" event={"ID":"f387b325-e109-4ea3-af39-59292992adb5","Type":"ContainerStarted","Data":"fabd32fa90b2f0ef05327d3e2e0c953bd5e2c2828f65ca6bc8f408c0977c533c"}
Sep 09 23:49:02.454185 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.454099    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr" event={"ID":"f387b325-e109-4ea3-af39-59292992adb5","Type":"ContainerStarted","Data":"5f6872492a85c71b28a16afec7e451b4f658c871171d7129a271c0425ad06845"}
Sep 09 23:49:02.454185 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.454128    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr" event={"ID":"f387b325-e109-4ea3-af39-59292992adb5","Type":"ContainerStarted","Data":"46471286151a6aab600929b33d51c09ed972a35aa73c0cbdc4e1d89e7da3441a"}
Sep 09 23:49:02.454185 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.454163    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr" event={"ID":"f387b325-e109-4ea3-af39-59292992adb5","Type":"ContainerStarted","Data":"d70405644a4942d94d577a13c1ca20548d916a2f5072c1f075f75d3cd13a64e7"}
Sep 09 23:49:02.455607 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.455576    2556 generic.go:356] "Generic (PLEG): container finished" podID="033a3818-5580-4cc4-9e17-a1d5fd0d6a71" containerID="983b125ba3173303d708d44a488a9e563f6a4f2ab994bffe15e1357d6ce47a4a" exitCode=0
Sep 09 23:49:02.455724 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.455641    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dr7db" event={"ID":"033a3818-5580-4cc4-9e17-a1d5fd0d6a71","Type":"ContainerDied","Data":"983b125ba3173303d708d44a488a9e563f6a4f2ab994bffe15e1357d6ce47a4a"}
Sep 09 23:49:02.457365 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.457337    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l72vc" event={"ID":"b62491be-c176-4206-9d7b-84caeae4c67d","Type":"ContainerStarted","Data":"7522bc347c7c2a36d4d1949c5742f5de619b00a63fcf79a6e7951879e0ad0fb9"}
Sep 09 23:49:02.459093 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.459070    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2w8q5" event={"ID":"0c59eba6-c8ab-4a3b-8c89-dad9d27a3922","Type":"ContainerStarted","Data":"313660bf12aea90fd43bee26cc303ba6dc9954c13d9c120e6162fe2ff5c822fe"}
Sep 09 23:49:02.460756 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.460736    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t" event={"ID":"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6","Type":"ContainerStarted","Data":"a5fc470942356f7d2a6402d465ab0de51a7518277ed94e7fdd4fed0bce8daae3"}
Sep 09 23:49:02.462403 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.462373    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l7btc" event={"ID":"4efb074d-6bb6-412a-8cfe-f70fcaaa1c5a","Type":"ContainerStarted","Data":"cfcfa21d76643ab86e306d5c327c6d26ce906a2ed2b4315473d34edf3e58a166"}
Sep 09 23:49:02.475796 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.475757    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7nmdr" podStartSLOduration=7.34891872 podStartE2EDuration="21.475743099s" totalImagesPullingTime="14.126824379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:02.459971813 +0000 UTC m=+22.711107856" watchObservedRunningTime="2026-09-09 23:49:02.475743099 +0000 UTC m=+22.726879144"
Sep 09 23:49:02.487754 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.487694    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wj527" podStartSLOduration=7.358793154 podStartE2EDuration="21.487680517s" totalImagesPullingTime="14.128887363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:02.475130722 +0000 UTC m=+22.726266766" watchObservedRunningTime="2026-09-09 23:49:02.487680517 +0000 UTC m=+22.738816618"
Sep 09 23:49:02.503700 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.503655    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l72vc" podStartSLOduration=3.514154962 podStartE2EDuration="21.503640742s" totalImagesPullingTime="17.98948578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:02.486938403 +0000 UTC m=+22.738074447" watchObservedRunningTime="2026-09-09 23:49:02.503640742 +0000 UTC m=+22.754776782"
Sep 09 23:49:02.519391 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.519345    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2w8q5" podStartSLOduration=3.469843041 podStartE2EDuration="21.519331792s" totalImagesPullingTime="18.049488751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:02.519298079 +0000 UTC m=+22.770434122" watchObservedRunningTime="2026-09-09 23:49:02.519331792 +0000 UTC m=+22.770467818"
Sep 09 23:49:02.761390 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:02.761356    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 23:49:03.232591 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.232501    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T23:49:02.761379985Z","UUID":"e6ef76ef-1018-46ff-b086-7b79a694b76a","Handler":null,"Name":"","Endpoint":""}
Sep 09 23:49:03.233909 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.233891    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 23:49:03.234018 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.233917    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 23:49:03.262322 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.262291    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:03.262472 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.262292    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:49:03.262557 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:03.262493    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:49:03.262710 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:03.262686    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:49:03.360689 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.360657    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:49:03.361434 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.361413    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:49:03.382362 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.382317    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l7btc" podStartSLOduration=4.388191012 podStartE2EDuration="22.38230236s" totalImagesPullingTime="17.994111348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:02.534264302 +0000 UTC m=+22.785400345" watchObservedRunningTime="2026-09-09 23:49:03.38230236 +0000 UTC m=+23.633438446"
Sep 09 23:49:03.466806 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.466775    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t" event={"ID":"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6","Type":"ContainerStarted","Data":"a34c0a82b3fe4f1f7e005f1b755b54c83c43e6c97eb59dd15459fa7d93820eaa"}
Sep 09 23:49:03.468163 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.468132    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6k6xm" event={"ID":"3cd21a19-cfa8-463f-87e8-b1fe25b0c247","Type":"ContainerStarted","Data":"58de5d548c2de6ebce6d4af4a9fa2654c021b8077b0264fbf28221913ebc0391"}
Sep 09 23:49:03.468959 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.468939    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:49:03.469136 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.468992    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7nmdr"
Sep 09 23:49:03.482343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:03.482281    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6k6xm" podStartSLOduration=4.495377644 podStartE2EDuration="22.48226201s" totalImagesPullingTime="17.986884366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:03.481853963 +0000 UTC m=+23.732990007" watchObservedRunningTime="2026-09-09 23:49:03.48226201 +0000 UTC m=+23.733398054"
Sep 09 23:49:04.262044 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:04.261955    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:04.262218 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:04.262090    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:49:04.473690 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:04.473624    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t" event={"ID":"0c5d38a2-3207-4a3b-923c-faa6dcd7fae6","Type":"ContainerStarted","Data":"cc0040414f4d3804e60103c81f9ded5103e934dd026c30bc5cfee8542a7aac28"}
Sep 09 23:49:04.478033 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:04.477993    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr" event={"ID":"f387b325-e109-4ea3-af39-59292992adb5","Type":"ContainerStarted","Data":"fadd3c8fd3eb6da633b638a2965e3dde7828d1a73462c471622600c8208e895b"}
Sep 09 23:49:05.262394 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:05.262359    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:49:05.262586 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:05.262374    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:05.262586 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:05.262522    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:49:05.262701 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:05.262598    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:49:06.261811 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:06.261777    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:06.262283 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:06.261925    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:49:07.261993 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:07.261819    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:07.262407 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:07.261873    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:49:07.262407 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:07.262095    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:49:07.262407 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:07.262202    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:49:08.262623 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:08.262584    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:08.263529 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:08.262733    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:49:08.490368 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:08.490333    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr" event={"ID":"f387b325-e109-4ea3-af39-59292992adb5","Type":"ContainerStarted","Data":"da5e80657058f5c1361f0473765aa1d84322134eb3805a22ddda4dabdf6cf9fa"}
Sep 09 23:49:08.490601 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:08.490583    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:49:08.492620 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:08.492465    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dr7db" event={"ID":"033a3818-5580-4cc4-9e17-a1d5fd0d6a71","Type":"ContainerStarted","Data":"ae40a4d54643782579a1af7a9319db3f9fa985bf5fc799f59911cb06579b061b"}
Sep 09 23:49:08.505818 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:08.505795    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:49:08.515938 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:08.515901    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr" podStartSLOduration=9.474218302 podStartE2EDuration="27.515888407s" totalImagesPullingTime="18.041670105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:08.515720121 +0000 UTC m=+28.766856164" watchObservedRunningTime="2026-09-09 23:49:08.515888407 +0000 UTC m=+28.767024450"
Sep 09 23:49:08.516326 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:08.516302    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wnj9t" podStartSLOduration=7.280691571 podStartE2EDuration="27.516295447s" totalImagesPullingTime="20.235603876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:04.490215818 +0000 UTC m=+24.741351861" watchObservedRunningTime="2026-09-09 23:49:08.516295447 +0000 UTC m=+28.767431594"
Sep 09 23:49:09.261740 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.261712    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:09.261907 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.261715    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:49:09.261907 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:09.261799    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:49:09.261907 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:09.261889    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:49:09.497032 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.496995    2556 generic.go:356] "Generic (PLEG): container finished" podID="033a3818-5580-4cc4-9e17-a1d5fd0d6a71" containerID="ae40a4d54643782579a1af7a9319db3f9fa985bf5fc799f59911cb06579b061b" exitCode=0
Sep 09 23:49:09.497518 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.497040    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dr7db" event={"ID":"033a3818-5580-4cc4-9e17-a1d5fd0d6a71","Type":"ContainerDied","Data":"ae40a4d54643782579a1af7a9319db3f9fa985bf5fc799f59911cb06579b061b"}
Sep 09 23:49:09.498427 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.497609    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:49:09.498427 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.497684    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:49:09.516562 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.516505    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:49:09.978514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.978386    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-plj9c"]
Sep 09 23:49:09.978744 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.978527    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:49:09.978744 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:09.978646    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:49:09.983163 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.983116    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kg65s"]
Sep 09 23:49:09.983289 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.983224    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:09.983354 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:09.983322    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:49:09.983748 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.983726    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bgd5b"]
Sep 09 23:49:09.983926 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:09.983813    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:09.983926 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:09.983891    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:49:11.262200 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:11.262164    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:11.262200 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:11.262187    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:11.262619 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:11.262164    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:49:11.262619 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:11.262299    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:49:11.262619 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:11.262418    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:49:11.262619 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:11.262501    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:49:11.503253 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:11.503223    2556 generic.go:356] "Generic (PLEG): container finished" podID="033a3818-5580-4cc4-9e17-a1d5fd0d6a71" containerID="c91d53ecdacc576a9a3fc3680985880a196493c896fad5c335d48fc377aaa3c3" exitCode=0
Sep 09 23:49:11.503389 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:11.503273    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dr7db" event={"ID":"033a3818-5580-4cc4-9e17-a1d5fd0d6a71","Type":"ContainerDied","Data":"c91d53ecdacc576a9a3fc3680985880a196493c896fad5c335d48fc377aaa3c3"}
Sep 09 23:49:13.262419 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:13.262206    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:49:13.262786 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:13.262211    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:13.262786 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:13.262445    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:49:13.262786 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:13.262250    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:13.262786 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:13.262530    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bgd5b" podUID="7f5d6c3d-1bac-4d49-9b92-7f64c87939ab"
Sep 09 23:49:13.262786 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:13.262588    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg65s" podUID="ab3c838b-f89a-4dd8-8318-705775e5bb2a"
Sep 09 23:49:13.509853 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:13.509819    2556 generic.go:356] "Generic (PLEG): container finished" podID="033a3818-5580-4cc4-9e17-a1d5fd0d6a71" containerID="ba1d61b64e6e4c4e2ae69ab6f76edd12ff0fe3b4e68de7b75f7572f06176e062" exitCode=0
Sep 09 23:49:13.510026 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:13.509865    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dr7db" event={"ID":"033a3818-5580-4cc4-9e17-a1d5fd0d6a71","Type":"ContainerDied","Data":"ba1d61b64e6e4c4e2ae69ab6f76edd12ff0fe3b4e68de7b75f7572f06176e062"}
Sep 09 23:49:13.928101 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:13.928069    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:49:13.928101 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:13.928105    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:13.928267 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:13.928200    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:49:13.928267 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:13.928214    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 23:49:13.928267 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:13.928264    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs podName:21369195-b9f6-4653-ae29-d2a7a07eb1e2 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:45.928249573 +0000 UTC m=+66.179385593 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs") pod "network-metrics-daemon-plj9c" (UID: "21369195-b9f6-4653-ae29-d2a7a07eb1e2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 23:49:13.928383 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:13.928285    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret podName:ab3c838b-f89a-4dd8-8318-705775e5bb2a nodeName:}" failed. No retries permitted until 2026-09-09 23:49:45.928275446 +0000 UTC m=+66.179411482 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret") pod "global-pull-secret-syncer-kg65s" (UID: "ab3c838b-f89a-4dd8-8318-705775e5bb2a") : object "kube-system"/"original-pull-secret" not registered
Sep 09 23:49:14.129980 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:14.129947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lf7x\" (UniqueName: \"kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x\") pod \"network-check-target-bgd5b\" (UID: \"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab\") " pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:14.130148 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:14.130079    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 23:49:14.130148 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:14.130095    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 23:49:14.130148 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:14.130104    2556 projected.go:196] Error preparing data for projected volume kube-api-access-7lf7x for pod openshift-network-diagnostics/network-check-target-bgd5b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:49:14.130257 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:14.130153    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x podName:7f5d6c3d-1bac-4d49-9b92-7f64c87939ab nodeName:}" failed. No retries permitted until 2026-09-09 23:49:46.13013875 +0000 UTC m=+66.381274771 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lf7x" (UniqueName: "kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x") pod "network-check-target-bgd5b" (UID: "7f5d6c3d-1bac-4d49-9b92-7f64c87939ab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 23:49:15.081345 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.081275    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-211.ec2.internal" event="NodeReady"
Sep 09 23:49:15.081743 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.081394    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 23:49:15.119241 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.119208    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-8jvvw"]
Sep 09 23:49:15.124751 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.124727    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-867bcc698d-spnq8"]
Sep 09 23:49:15.124898 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.124879    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:15.127233 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.127212    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mqnps\""
Sep 09 23:49:15.127338 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.127319    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 23:49:15.127695 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.127672    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 23:49:15.129601 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.129562    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.130762 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.130742    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zz486"]
Sep 09 23:49:15.131608 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.131582    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 23:49:15.131727 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.131657    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 23:49:15.131727 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.131722    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 23:49:15.131943 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.131923    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rr4p2\""
Sep 09 23:49:15.137235 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.136107    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ctkq2"]
Sep 09 23:49:15.137235 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.136242    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.138193 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.138170    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 23:49:15.139245 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.139230    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 23:49:15.139794 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.139776    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 23:49:15.139873 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.139858    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j4rs2\""
Sep 09 23:49:15.143486 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.143470    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ctkq2"
Sep 09 23:49:15.143486 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.143483    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-8jvvw"]
Sep 09 23:49:15.145909 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.145556    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-867bcc698d-spnq8"]
Sep 09 23:49:15.145909 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.145902    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kqlfp\""
Sep 09 23:49:15.146359 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.146341    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 23:49:15.146452 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.146397    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 23:49:15.146638 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.146622    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 23:49:15.153502 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.153481    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ctkq2"]
Sep 09 23:49:15.160433 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.160389    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zz486"]
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.237751    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96cde171-d33b-4082-b727-bfd2d2574b7e-cert\") pod \"ingress-canary-ctkq2\" (UID: \"96cde171-d33b-4082-b727-bfd2d2574b7e\") " pod="openshift-ingress-canary/ingress-canary-ctkq2"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.237797    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrd2v\" (UniqueName: \"kubernetes.io/projected/96cde171-d33b-4082-b727-bfd2d2574b7e-kube-api-access-vrd2v\") pod \"ingress-canary-ctkq2\" (UID: \"96cde171-d33b-4082-b727-bfd2d2574b7e\") " pod="openshift-ingress-canary/ingress-canary-ctkq2"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.237831    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.237862    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d59d7cbb-123e-4203-907e-58c96c6ef1f9-ca-trust-extracted\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.237890    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznrr\" (UniqueName: \"kubernetes.io/projected/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-kube-api-access-qznrr\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.237922    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.237944    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-trusted-ca\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.237971    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.238009    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-bound-sa-token\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.238037    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-config-volume\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.238085    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-tmp-dir\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.238112    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-installation-pull-secrets\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.238162    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-image-registry-private-configuration\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.238225    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-certificates\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.239797 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.238266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6543ed96-cfa4-4666-8547-ec6fd01de0c1-nginx-conf\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:15.240604 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.238297    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52jm5\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-kube-api-access-52jm5\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.262329 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.262295    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:15.262470 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.262366    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:15.262470 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.262379    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:49:15.264835 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.264815    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 23:49:15.265019 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.265001    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 23:49:15.265105 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.265042    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 23:49:15.265224 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.265202    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 23:49:15.265333 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.265278    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-df57b\""
Sep 09 23:49:15.265425 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.265401    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lqch4\""
Sep 09 23:49:15.339417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339336    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96cde171-d33b-4082-b727-bfd2d2574b7e-cert\") pod \"ingress-canary-ctkq2\" (UID: \"96cde171-d33b-4082-b727-bfd2d2574b7e\") " pod="openshift-ingress-canary/ingress-canary-ctkq2"
Sep 09 23:49:15.339417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339378    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrd2v\" (UniqueName: \"kubernetes.io/projected/96cde171-d33b-4082-b727-bfd2d2574b7e-kube-api-access-vrd2v\") pod \"ingress-canary-ctkq2\" (UID: \"96cde171-d33b-4082-b727-bfd2d2574b7e\") " pod="openshift-ingress-canary/ingress-canary-ctkq2"
Sep 09 23:49:15.339417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339400    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:15.339417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339420    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d59d7cbb-123e-4203-907e-58c96c6ef1f9-ca-trust-extracted\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.339714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339436    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznrr\" (UniqueName: \"kubernetes.io/projected/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-kube-api-access-qznrr\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.339714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339461    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.339714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339486    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-trusted-ca\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.339714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339508    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.339714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-bound-sa-token\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.339714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339561    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-config-volume\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.339714 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.339584    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 23:49:15.339714 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.339670    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert podName:6543ed96-cfa4-4666-8547-ec6fd01de0c1 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:15.839648621 +0000 UTC m=+36.090784657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8jvvw" (UID: "6543ed96-cfa4-4666-8547-ec6fd01de0c1") : secret "networking-console-plugin-cert" not found
Sep 09 23:49:15.340070 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339856    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d59d7cbb-123e-4203-907e-58c96c6ef1f9-ca-trust-extracted\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.340070 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.339947    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 23:49:15.340070 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.339967    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867bcc698d-spnq8: secret "image-registry-tls" not found
Sep 09 23:49:15.340070 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339589    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-tmp-dir\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.340070 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.340020    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls podName:d59d7cbb-123e-4203-907e-58c96c6ef1f9 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:15.840004843 +0000 UTC m=+36.091140880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls") pod "image-registry-867bcc698d-spnq8" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9") : secret "image-registry-tls" not found
Sep 09 23:49:15.340070 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.339860    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-tmp-dir\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.340380 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.340076    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-installation-pull-secrets\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.340380 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.340120    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-image-registry-private-configuration\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.340380 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.340183    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-certificates\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.340380 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.340229    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6543ed96-cfa4-4666-8547-ec6fd01de0c1-nginx-conf\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:15.340380 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.340240    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:15.340380 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.340254    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52jm5\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-kube-api-access-52jm5\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.340380 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.340298    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls podName:a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:15.840283855 +0000 UTC m=+36.091419899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls") pod "dns-default-zz486" (UID: "a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:15.340727 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.340704    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-config-volume\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.341114 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.341088    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-trusted-ca\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.341114 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.341103    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6543ed96-cfa4-4666-8547-ec6fd01de0c1-nginx-conf\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:15.341264 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.341186    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-certificates\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.344160 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.344138    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-installation-pull-secrets\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.344270 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.344159    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-image-registry-private-configuration\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.344270 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.344209    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96cde171-d33b-4082-b727-bfd2d2574b7e-cert\") pod \"ingress-canary-ctkq2\" (UID: \"96cde171-d33b-4082-b727-bfd2d2574b7e\") " pod="openshift-ingress-canary/ingress-canary-ctkq2"
Sep 09 23:49:15.348231 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.348188    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznrr\" (UniqueName: \"kubernetes.io/projected/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-kube-api-access-qznrr\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.348524 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.348498    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-bound-sa-token\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.348707 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.348688    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52jm5\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-kube-api-access-52jm5\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.349027 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.349011    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrd2v\" (UniqueName: \"kubernetes.io/projected/96cde171-d33b-4082-b727-bfd2d2574b7e-kube-api-access-vrd2v\") pod \"ingress-canary-ctkq2\" (UID: \"96cde171-d33b-4082-b727-bfd2d2574b7e\") " pod="openshift-ingress-canary/ingress-canary-ctkq2"
Sep 09 23:49:15.461845 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.461814    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ctkq2"
Sep 09 23:49:15.763047 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.763015    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ctkq2"]
Sep 09 23:49:15.767112 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:49:15.767087    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cde171_d33b_4082_b727_bfd2d2574b7e.slice/crio-c0686d6663becad242eccefc2fe77ad48889ee77c6e3ad22003de7e4b5da4455 WatchSource:0}: Error finding container c0686d6663becad242eccefc2fe77ad48889ee77c6e3ad22003de7e4b5da4455: Status 404 returned error can't find the container with id c0686d6663becad242eccefc2fe77ad48889ee77c6e3ad22003de7e4b5da4455
Sep 09 23:49:15.844674 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.844645    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:15.844816 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.844688    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:15.844816 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:15.844707    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:15.844816 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.844803    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:15.844816 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.844807    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 23:49:15.845002 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.844828    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867bcc698d-spnq8: secret "image-registry-tls" not found
Sep 09 23:49:15.845002 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.844803    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 23:49:15.845002 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.844869    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls podName:a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:16.844841302 +0000 UTC m=+37.095977337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls") pod "dns-default-zz486" (UID: "a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:15.845002 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.844909    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls podName:d59d7cbb-123e-4203-907e-58c96c6ef1f9 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:16.844896854 +0000 UTC m=+37.096032885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls") pod "image-registry-867bcc698d-spnq8" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9") : secret "image-registry-tls" not found
Sep 09 23:49:15.845002 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:15.844923    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert podName:6543ed96-cfa4-4666-8547-ec6fd01de0c1 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:16.84491463 +0000 UTC m=+37.096050651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8jvvw" (UID: "6543ed96-cfa4-4666-8547-ec6fd01de0c1") : secret "networking-console-plugin-cert" not found
Sep 09 23:49:16.519679 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:16.519638    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ctkq2" event={"ID":"96cde171-d33b-4082-b727-bfd2d2574b7e","Type":"ContainerStarted","Data":"c0686d6663becad242eccefc2fe77ad48889ee77c6e3ad22003de7e4b5da4455"}
Sep 09 23:49:16.854016 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:16.853779    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:16.854185 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:16.854014    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:16.854185 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:16.854040    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:16.854185 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:16.853935    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 23:49:16.854185 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:16.854150    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert podName:6543ed96-cfa4-4666-8547-ec6fd01de0c1 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:18.854128193 +0000 UTC m=+39.105264232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8jvvw" (UID: "6543ed96-cfa4-4666-8547-ec6fd01de0c1") : secret "networking-console-plugin-cert" not found
Sep 09 23:49:16.854185 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:16.854162    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:16.854436 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:16.854213    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls podName:a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:18.854201434 +0000 UTC m=+39.105337468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls") pod "dns-default-zz486" (UID: "a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:16.854436 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:16.854166    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 23:49:16.854436 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:16.854232    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867bcc698d-spnq8: secret "image-registry-tls" not found
Sep 09 23:49:16.854436 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:16.854271    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls podName:d59d7cbb-123e-4203-907e-58c96c6ef1f9 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:18.85426036 +0000 UTC m=+39.105396400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls") pod "image-registry-867bcc698d-spnq8" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9") : secret "image-registry-tls" not found
Sep 09 23:49:18.869216 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:18.869137    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:18.869680 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:18.869275    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 23:49:18.869680 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:18.869302    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:18.869680 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:18.869345    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert podName:6543ed96-cfa4-4666-8547-ec6fd01de0c1 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:22.869325021 +0000 UTC m=+43.120461065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8jvvw" (UID: "6543ed96-cfa4-4666-8547-ec6fd01de0c1") : secret "networking-console-plugin-cert" not found
Sep 09 23:49:18.869680 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:18.869369    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:18.869680 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:18.869426    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 23:49:18.869680 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:18.869443    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867bcc698d-spnq8: secret "image-registry-tls" not found
Sep 09 23:49:18.869680 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:18.869458    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:18.869680 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:18.869499    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls podName:d59d7cbb-123e-4203-907e-58c96c6ef1f9 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:22.869482138 +0000 UTC m=+43.120618159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls") pod "image-registry-867bcc698d-spnq8" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9") : secret "image-registry-tls" not found
Sep 09 23:49:18.869680 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:18.869517    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls podName:a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:22.869508865 +0000 UTC m=+43.120644890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls") pod "dns-default-zz486" (UID: "a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:20.531543 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:20.531513    2556 generic.go:356] "Generic (PLEG): container finished" podID="033a3818-5580-4cc4-9e17-a1d5fd0d6a71" containerID="c0693665a7b730b9c43d7ff31deba4125ae814fc2ffc974fb8e9d0eb54acd80d" exitCode=0
Sep 09 23:49:20.531974 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:20.531597    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dr7db" event={"ID":"033a3818-5580-4cc4-9e17-a1d5fd0d6a71","Type":"ContainerDied","Data":"c0693665a7b730b9c43d7ff31deba4125ae814fc2ffc974fb8e9d0eb54acd80d"}
Sep 09 23:49:20.533077 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:20.533040    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ctkq2" event={"ID":"96cde171-d33b-4082-b727-bfd2d2574b7e","Type":"ContainerStarted","Data":"54df40368e2624e6171311265973a6bc159a54774087d5cd0a81458d2056c4cb"}
Sep 09 23:49:20.566932 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:20.566891    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ctkq2" podStartSLOduration=1.715947319 podStartE2EDuration="5.566877885s" totalImagesPullingTime="3.850930566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:20.566178571 +0000 UTC m=+40.817314614" watchObservedRunningTime="2026-09-09 23:49:20.566877885 +0000 UTC m=+40.818013928"
Sep 09 23:49:21.538395 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:21.538360    2556 generic.go:356] "Generic (PLEG): container finished" podID="033a3818-5580-4cc4-9e17-a1d5fd0d6a71" containerID="3d92042daec2e7ab16c22a63c087f3d1fd6609c5dd2b3d6949e192e40a2f8c32" exitCode=0
Sep 09 23:49:21.538778 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:21.538435    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dr7db" event={"ID":"033a3818-5580-4cc4-9e17-a1d5fd0d6a71","Type":"ContainerDied","Data":"3d92042daec2e7ab16c22a63c087f3d1fd6609c5dd2b3d6949e192e40a2f8c32"}
Sep 09 23:49:22.543665 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:22.543634    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dr7db" event={"ID":"033a3818-5580-4cc4-9e17-a1d5fd0d6a71","Type":"ContainerStarted","Data":"6cc56b43423cf84d0b08c20968eb3a2779e181b42494f8f398ce51c0d84f4004"}
Sep 09 23:49:22.565522 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:22.565478    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dr7db" podStartSLOduration=9.415064617 podStartE2EDuration="41.565463536s" totalImagesPullingTime="32.150398919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:22.563867269 +0000 UTC m=+42.815003313" watchObservedRunningTime="2026-09-09 23:49:22.565463536 +0000 UTC m=+42.816599576"
Sep 09 23:49:22.896609 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:22.896542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:22.896609 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:22.896576    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:22.896609 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:22.896595    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:22.896802 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:22.896679    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:22.896802 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:22.896694    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 23:49:22.896802 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:22.896715    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867bcc698d-spnq8: secret "image-registry-tls" not found
Sep 09 23:49:22.896802 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:22.896721    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls podName:a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:30.896709722 +0000 UTC m=+51.147845743 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls") pod "dns-default-zz486" (UID: "a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:22.896802 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:22.896760    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 23:49:22.896802 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:22.896764    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls podName:d59d7cbb-123e-4203-907e-58c96c6ef1f9 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:30.896748374 +0000 UTC m=+51.147884404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls") pod "image-registry-867bcc698d-spnq8" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9") : secret "image-registry-tls" not found
Sep 09 23:49:22.896802 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:22.896807    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert podName:6543ed96-cfa4-4666-8547-ec6fd01de0c1 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:30.896798536 +0000 UTC m=+51.147934557 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8jvvw" (UID: "6543ed96-cfa4-4666-8547-ec6fd01de0c1") : secret "networking-console-plugin-cert" not found
Sep 09 23:49:30.954406 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:30.954350    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:30.954406 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:30.954406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:30.954406 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:30.954425    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:30.954857 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:30.954494    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 23:49:30.954857 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:30.954509    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 23:49:30.954857 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:30.954534    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867bcc698d-spnq8: secret "image-registry-tls" not found
Sep 09 23:49:30.954857 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:30.954554    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert podName:6543ed96-cfa4-4666-8547-ec6fd01de0c1 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:46.954539324 +0000 UTC m=+67.205675350 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8jvvw" (UID: "6543ed96-cfa4-4666-8547-ec6fd01de0c1") : secret "networking-console-plugin-cert" not found
Sep 09 23:49:30.954857 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:30.954589    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls podName:d59d7cbb-123e-4203-907e-58c96c6ef1f9 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:46.954576217 +0000 UTC m=+67.205712238 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls") pod "image-registry-867bcc698d-spnq8" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9") : secret "image-registry-tls" not found
Sep 09 23:49:30.954857 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:30.954518    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:30.954857 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:30.954622    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls podName:a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2 nodeName:}" failed. No retries permitted until 2026-09-09 23:49:46.954613829 +0000 UTC m=+67.205749855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls") pod "dns-default-zz486" (UID: "a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:41.518249 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:41.518220    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k9kxr"
Sep 09 23:49:45.958485 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:45.958435    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:49:45.958485 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:45.958489    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:45.960636 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:45.960617    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 23:49:45.960828 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:45.960812    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 23:49:45.969444 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:45.969430    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 23:49:45.969537 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:45.969486    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs podName:21369195-b9f6-4653-ae29-d2a7a07eb1e2 nodeName:}" failed. No retries permitted until 2026-09-09 23:50:49.969469136 +0000 UTC m=+130.220605157 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs") pod "network-metrics-daemon-plj9c" (UID: "21369195-b9f6-4653-ae29-d2a7a07eb1e2") : secret "metrics-daemon-secret" not found
Sep 09 23:49:45.972166 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:45.972151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab3c838b-f89a-4dd8-8318-705775e5bb2a-original-pull-secret\") pod \"global-pull-secret-syncer-kg65s\" (UID: \"ab3c838b-f89a-4dd8-8318-705775e5bb2a\") " pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:46.159574 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.159523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lf7x\" (UniqueName: \"kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x\") pod \"network-check-target-bgd5b\" (UID: \"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab\") " pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:46.161941 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.161925    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 23:49:46.172543 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.172526    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 23:49:46.173601 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.173589    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg65s"
Sep 09 23:49:46.183417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.183397    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lf7x\" (UniqueName: \"kubernetes.io/projected/7f5d6c3d-1bac-4d49-9b92-7f64c87939ab-kube-api-access-7lf7x\") pod \"network-check-target-bgd5b\" (UID: \"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab\") " pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:46.438150 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.438120    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kg65s"]
Sep 09 23:49:46.441275 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:49:46.441248    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3c838b_f89a_4dd8_8318_705775e5bb2a.slice/crio-fa4987155d818818a07fe35e56b2221911cac56fd00a3eea5acbfc4b296a057f WatchSource:0}: Error finding container fa4987155d818818a07fe35e56b2221911cac56fd00a3eea5acbfc4b296a057f: Status 404 returned error can't find the container with id fa4987155d818818a07fe35e56b2221911cac56fd00a3eea5acbfc4b296a057f
Sep 09 23:49:46.483074 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.482981    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-df57b\""
Sep 09 23:49:46.490973 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.490950    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:46.599576 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.599544    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kg65s" event={"ID":"ab3c838b-f89a-4dd8-8318-705775e5bb2a","Type":"ContainerStarted","Data":"fa4987155d818818a07fe35e56b2221911cac56fd00a3eea5acbfc4b296a057f"}
Sep 09 23:49:46.754461 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.754386    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bgd5b"]
Sep 09 23:49:46.757145 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:49:46.757117    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5d6c3d_1bac_4d49_9b92_7f64c87939ab.slice/crio-d11c87de4d5be389a1f1f6b5d6de823861f36e817421657a86140999be86bf66 WatchSource:0}: Error finding container d11c87de4d5be389a1f1f6b5d6de823861f36e817421657a86140999be86bf66: Status 404 returned error can't find the container with id d11c87de4d5be389a1f1f6b5d6de823861f36e817421657a86140999be86bf66
Sep 09 23:49:46.966832 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.966794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:49:46.967321 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.966847    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:49:46.967321 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:46.966898    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:49:46.967321 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:46.966938    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 23:49:46.967321 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:46.967003    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 23:49:46.967321 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:46.967009    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:49:46.967321 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:46.967019    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867bcc698d-spnq8: secret "image-registry-tls" not found
Sep 09 23:49:46.967321 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:46.967035    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert podName:6543ed96-cfa4-4666-8547-ec6fd01de0c1 nodeName:}" failed. No retries permitted until 2026-09-09 23:50:18.967011293 +0000 UTC m=+99.218147328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8jvvw" (UID: "6543ed96-cfa4-4666-8547-ec6fd01de0c1") : secret "networking-console-plugin-cert" not found
Sep 09 23:49:46.967321 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:46.967079    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls podName:a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2 nodeName:}" failed. No retries permitted until 2026-09-09 23:50:18.967044947 +0000 UTC m=+99.218180971 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls") pod "dns-default-zz486" (UID: "a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2") : secret "dns-default-metrics-tls" not found
Sep 09 23:49:46.967321 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:49:46.967100    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls podName:d59d7cbb-123e-4203-907e-58c96c6ef1f9 nodeName:}" failed. No retries permitted until 2026-09-09 23:50:18.967088899 +0000 UTC m=+99.218224925 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls") pod "image-registry-867bcc698d-spnq8" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9") : secret "image-registry-tls" not found
Sep 09 23:49:47.603743 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:47.603702    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bgd5b" event={"ID":"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab","Type":"ContainerStarted","Data":"d11c87de4d5be389a1f1f6b5d6de823861f36e817421657a86140999be86bf66"}
Sep 09 23:49:51.616456 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:51.616365    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kg65s" event={"ID":"ab3c838b-f89a-4dd8-8318-705775e5bb2a","Type":"ContainerStarted","Data":"22fc1bbde48762994f9d994247856908eca0ae5d73059186625d7a7697cbfb91"}
Sep 09 23:49:51.617943 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:51.617917    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bgd5b" event={"ID":"7f5d6c3d-1bac-4d49-9b92-7f64c87939ab","Type":"ContainerStarted","Data":"5b2dc5d4af9c308dab6344f010e2538453b2fad283e39aff605c84911fcd9520"}
Sep 09 23:49:51.618090 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:51.618069    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:49:51.629449 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:51.629411    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kg65s" podStartSLOduration=65.803534335 podStartE2EDuration="1m10.629400993s" totalImagesPullingTime="4.825866658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:51.628864707 +0000 UTC m=+71.880000750" watchObservedRunningTime="2026-09-09 23:49:51.629400993 +0000 UTC m=+71.880537037"
Sep 09 23:49:51.644904 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:49:51.644868    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bgd5b" podStartSLOduration=66.138368413 podStartE2EDuration="1m10.644857879s" totalImagesPullingTime="4.506489466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:49:51.643849669 +0000 UTC m=+71.894985709" watchObservedRunningTime="2026-09-09 23:49:51.644857879 +0000 UTC m=+71.895993912"
Sep 09 23:50:18.984296 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:18.984148    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:50:18.984296 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:18.984208    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:50:18.984296 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:18.984227    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:50:18.984884 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:50:18.984302    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 23:50:18.984884 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:50:18.984308    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:50:18.984884 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:50:18.984369    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 23:50:18.984884 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:50:18.984387    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867bcc698d-spnq8: secret "image-registry-tls" not found
Sep 09 23:50:18.984884 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:50:18.984391    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert podName:6543ed96-cfa4-4666-8547-ec6fd01de0c1 nodeName:}" failed. No retries permitted until 2026-09-09 23:51:22.984367559 +0000 UTC m=+163.235503600 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8jvvw" (UID: "6543ed96-cfa4-4666-8547-ec6fd01de0c1") : secret "networking-console-plugin-cert" not found
Sep 09 23:50:18.984884 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:50:18.984470    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls podName:a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2 nodeName:}" failed. No retries permitted until 2026-09-09 23:51:22.984452409 +0000 UTC m=+163.235588444 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls") pod "dns-default-zz486" (UID: "a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2") : secret "dns-default-metrics-tls" not found
Sep 09 23:50:18.984884 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:50:18.984485    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls podName:d59d7cbb-123e-4203-907e-58c96c6ef1f9 nodeName:}" failed. No retries permitted until 2026-09-09 23:51:22.984476716 +0000 UTC m=+163.235612738 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls") pod "image-registry-867bcc698d-spnq8" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9") : secret "image-registry-tls" not found
Sep 09 23:50:22.622990 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:22.622955    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bgd5b"
Sep 09 23:50:26.782131 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.782084    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"]
Sep 09 23:50:26.787020 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.786984    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:26.789759 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.789728    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 23:50:26.790790 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.790764    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 23:50:26.790923 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.790815    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 23:50:26.790923 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.790873    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 23:50:26.794146 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.794114    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"]
Sep 09 23:50:26.836368 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.836337    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb-klusterlet-config\") pod \"klusterlet-addon-workmgr-5dc86fd58b-2q8l9\" (UID: \"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:26.836472 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.836383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc994\" (UniqueName: \"kubernetes.io/projected/d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb-kube-api-access-sc994\") pod \"klusterlet-addon-workmgr-5dc86fd58b-2q8l9\" (UID: \"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:26.836472 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.836462    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb-tmp\") pod \"klusterlet-addon-workmgr-5dc86fd58b-2q8l9\" (UID: \"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:26.937151 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.937121    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb-tmp\") pod \"klusterlet-addon-workmgr-5dc86fd58b-2q8l9\" (UID: \"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:26.937251 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.937173    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb-klusterlet-config\") pod \"klusterlet-addon-workmgr-5dc86fd58b-2q8l9\" (UID: \"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:26.937251 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.937204    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc994\" (UniqueName: \"kubernetes.io/projected/d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb-kube-api-access-sc994\") pod \"klusterlet-addon-workmgr-5dc86fd58b-2q8l9\" (UID: \"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:26.937516 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.937497    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb-tmp\") pod \"klusterlet-addon-workmgr-5dc86fd58b-2q8l9\" (UID: \"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:26.939889 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.939854    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb-klusterlet-config\") pod \"klusterlet-addon-workmgr-5dc86fd58b-2q8l9\" (UID: \"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:26.944822 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:26.944794    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc994\" (UniqueName: \"kubernetes.io/projected/d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb-kube-api-access-sc994\") pod \"klusterlet-addon-workmgr-5dc86fd58b-2q8l9\" (UID: \"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:27.097132 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:27.097075    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:27.363216 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:27.363139    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"]
Sep 09 23:50:27.366302 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:50:27.366261    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13cdbee_8d67_4e81_a45b_e74ffb3f5ebb.slice/crio-e960b47195731d39334157abcf4219096b71397ee15b8b7ef7e80194a12456f2 WatchSource:0}: Error finding container e960b47195731d39334157abcf4219096b71397ee15b8b7ef7e80194a12456f2: Status 404 returned error can't find the container with id e960b47195731d39334157abcf4219096b71397ee15b8b7ef7e80194a12456f2
Sep 09 23:50:27.700616 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:27.700545    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9" event={"ID":"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb","Type":"ContainerStarted","Data":"e960b47195731d39334157abcf4219096b71397ee15b8b7ef7e80194a12456f2"}
Sep 09 23:50:31.712791 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:31.712761    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9" event={"ID":"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb","Type":"ContainerStarted","Data":"ccca15487a76da56367da1f6ed36bcfafb2118d2f81bbdd82ae3a272f136931d"}
Sep 09 23:50:31.713182 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:31.712989    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:31.714348 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:31.714328    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:50:31.727546 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:31.727497    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9" podStartSLOduration=1.7845690539999999 podStartE2EDuration="5.727482757s" totalImagesPullingTime="3.942913703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:50:31.726973264 +0000 UTC m=+111.978109307" watchObservedRunningTime="2026-09-09 23:50:31.727482757 +0000 UTC m=+111.978618801"
Sep 09 23:50:49.999802 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:50:49.999763    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:50:50.000245 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:50:49.999874    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 23:50:50.000245 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:50:49.999938    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs podName:21369195-b9f6-4653-ae29-d2a7a07eb1e2 nodeName:}" failed. No retries permitted until 2026-09-09 23:52:51.999923222 +0000 UTC m=+252.251059256 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs") pod "network-metrics-daemon-plj9c" (UID: "21369195-b9f6-4653-ae29-d2a7a07eb1e2") : secret "metrics-daemon-secret" not found
Sep 09 23:51:18.138216 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:51:18.138166    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw" podUID="6543ed96-cfa4-4666-8547-ec6fd01de0c1"
Sep 09 23:51:18.147313 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:51:18.147286    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-867bcc698d-spnq8" podUID="d59d7cbb-123e-4203-907e-58c96c6ef1f9"
Sep 09 23:51:18.155448 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:51:18.155428    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zz486" podUID="a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2"
Sep 09 23:51:18.286933 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:51:18.286900    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:51:18.828002 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:51:18.827970    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zz486"
Sep 09 23:51:18.828213 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:51:18.827970    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:51:18.828213 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:51:18.827971    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:51:23.018103 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:51:23.018037    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:51:23.018103 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:51:23.018101    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:51:23.018527 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:51:23.018121    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:51:23.018527 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:51:23.018195    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 23:51:23.018527 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:51:23.018220    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 23:51:23.018527 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:51:23.018242    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 23:51:23.018527 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:51:23.018256    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-867bcc698d-spnq8: secret "image-registry-tls" not found
Sep 09 23:51:23.018527 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:51:23.018284    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert podName:6543ed96-cfa4-4666-8547-ec6fd01de0c1 nodeName:}" failed. No retries permitted until 2026-09-09 23:53:25.018262049 +0000 UTC m=+285.269398069 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8jvvw" (UID: "6543ed96-cfa4-4666-8547-ec6fd01de0c1") : secret "networking-console-plugin-cert" not found
Sep 09 23:51:23.018527 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:51:23.018303    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls podName:d59d7cbb-123e-4203-907e-58c96c6ef1f9 nodeName:}" failed. No retries permitted until 2026-09-09 23:53:25.018293725 +0000 UTC m=+285.269429745 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls") pod "image-registry-867bcc698d-spnq8" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9") : secret "image-registry-tls" not found
Sep 09 23:51:23.018527 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:51:23.018317    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls podName:a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2 nodeName:}" failed. No retries permitted until 2026-09-09 23:53:25.018309563 +0000 UTC m=+285.269445584 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls") pod "dns-default-zz486" (UID: "a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2") : secret "dns-default-metrics-tls" not found
Sep 09 23:51:30.263228 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:51:30.263199    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:52:31.713969 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:31.713921    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9" podUID="d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused"
Sep 09 23:52:32.003568 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:32.003480    2556 generic.go:356] "Generic (PLEG): container finished" podID="d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb" containerID="ccca15487a76da56367da1f6ed36bcfafb2118d2f81bbdd82ae3a272f136931d" exitCode=1
Sep 09 23:52:32.003568 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:32.003520    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9" event={"ID":"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb","Type":"ContainerDied","Data":"ccca15487a76da56367da1f6ed36bcfafb2118d2f81bbdd82ae3a272f136931d"}
Sep 09 23:52:32.003832 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:32.003817    2556 scope.go:117] "RemoveContainer" containerID="ccca15487a76da56367da1f6ed36bcfafb2118d2f81bbdd82ae3a272f136931d"
Sep 09 23:52:33.008131 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:33.008099    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9" event={"ID":"d13cdbee-8d67-4e81-a45b-e74ffb3f5ebb","Type":"ContainerStarted","Data":"eedccf9a87120a1b60733c94560dc915c40a13ff300bf4c65cdc4749a6f08528"}
Sep 09 23:52:33.008520 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:33.008380    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:52:33.008984 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:33.008967    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5dc86fd58b-2q8l9"
Sep 09 23:52:52.003605 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.003566    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:52:52.004109 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:52.003677    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 23:52:52.004109 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:52.003734    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs podName:21369195-b9f6-4653-ae29-d2a7a07eb1e2 nodeName:}" failed. No retries permitted until 2026-09-09 23:54:54.003717766 +0000 UTC m=+374.254853787 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs") pod "network-metrics-daemon-plj9c" (UID: "21369195-b9f6-4653-ae29-d2a7a07eb1e2") : secret "metrics-daemon-secret" not found
Sep 09 23:52:52.068704 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.068677    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-j86lt"]
Sep 09 23:52:52.071624 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.071594    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-j86lt"
Sep 09 23:52:52.074101 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.074080    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9q6sn\""
Sep 09 23:52:52.086245 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.086223    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-j86lt"]
Sep 09 23:52:52.173788 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.173757    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"]
Sep 09 23:52:52.176644 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.176627    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"]
Sep 09 23:52:52.176777 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.176758    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:52:52.179351 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.179320    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 23:52:52.179439 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.179351    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:52.179563 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.179545    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 23:52:52.179927 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.179901    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 23:52:52.180223 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.180194    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qschh\""
Sep 09 23:52:52.181806 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.181781    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dscd2\""
Sep 09 23:52:52.181925 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.181904    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 23:52:52.182129 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.182106    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 23:52:52.183341 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.183310    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 23:52:52.186511 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.185484    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 23:52:52.191654 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.191631    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"]
Sep 09 23:52:52.192750 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.192728    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"]
Sep 09 23:52:52.204268 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.204245    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbd4p\" (UniqueName: \"kubernetes.io/projected/c1873ca0-2b52-4016-b708-cde625b04c31-kube-api-access-dbd4p\") pod \"network-check-source-685456655f-j86lt\" (UID: \"c1873ca0-2b52-4016-b708-cde625b04c31\") " pod="openshift-network-diagnostics/network-check-source-685456655f-j86lt"
Sep 09 23:52:52.305017 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.304966    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbd4p\" (UniqueName: \"kubernetes.io/projected/c1873ca0-2b52-4016-b708-cde625b04c31-kube-api-access-dbd4p\") pod \"network-check-source-685456655f-j86lt\" (UID: \"c1873ca0-2b52-4016-b708-cde625b04c31\") " pod="openshift-network-diagnostics/network-check-source-685456655f-j86lt"
Sep 09 23:52:52.305017 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.304994    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q8vzf\" (UID: \"c2cf5466-b98c-49aa-91b1-eb36d07604e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:52:52.305017 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.305015    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:52.305226 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.305031    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:52.305226 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.305137    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmb4z\" (UniqueName: \"kubernetes.io/projected/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-kube-api-access-hmb4z\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:52.305226 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.305181    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqv4\" (UniqueName: \"kubernetes.io/projected/c2cf5466-b98c-49aa-91b1-eb36d07604e1-kube-api-access-2hqv4\") pod \"cluster-samples-operator-7cdcfcf76c-q8vzf\" (UID: \"c2cf5466-b98c-49aa-91b1-eb36d07604e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:52:52.312124 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.312104    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbd4p\" (UniqueName: \"kubernetes.io/projected/c1873ca0-2b52-4016-b708-cde625b04c31-kube-api-access-dbd4p\") pod \"network-check-source-685456655f-j86lt\" (UID: \"c1873ca0-2b52-4016-b708-cde625b04c31\") " pod="openshift-network-diagnostics/network-check-source-685456655f-j86lt"
Sep 09 23:52:52.380316 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.380293    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-j86lt"
Sep 09 23:52:52.405763 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.405741    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqv4\" (UniqueName: \"kubernetes.io/projected/c2cf5466-b98c-49aa-91b1-eb36d07604e1-kube-api-access-2hqv4\") pod \"cluster-samples-operator-7cdcfcf76c-q8vzf\" (UID: \"c2cf5466-b98c-49aa-91b1-eb36d07604e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:52:52.405883 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.405804    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q8vzf\" (UID: \"c2cf5466-b98c-49aa-91b1-eb36d07604e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:52:52.405883 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.405842    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:52.405994 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.405867    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:52.405994 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:52.405927    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 23:52:52.405994 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.405947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmb4z\" (UniqueName: \"kubernetes.io/projected/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-kube-api-access-hmb4z\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:52.406202 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:52.406011    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls podName:c2cf5466-b98c-49aa-91b1-eb36d07604e1 nodeName:}" failed. No retries permitted until 2026-09-09 23:52:52.905988774 +0000 UTC m=+253.157124810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-q8vzf" (UID: "c2cf5466-b98c-49aa-91b1-eb36d07604e1") : secret "samples-operator-tls" not found
Sep 09 23:52:52.406202 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:52.406038    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 23:52:52.406202 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:52.406121    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls podName:012fd72d-5acd-4896-9a66-ad26dbdc2a4e nodeName:}" failed. No retries permitted until 2026-09-09 23:52:52.906106369 +0000 UTC m=+253.157242403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-m6xqq" (UID: "012fd72d-5acd-4896-9a66-ad26dbdc2a4e") : secret "cluster-monitoring-operator-tls" not found
Sep 09 23:52:52.406562 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.406543    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:52.415979 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.415934    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmb4z\" (UniqueName: \"kubernetes.io/projected/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-kube-api-access-hmb4z\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:52.416333 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.416309    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqv4\" (UniqueName: \"kubernetes.io/projected/c2cf5466-b98c-49aa-91b1-eb36d07604e1-kube-api-access-2hqv4\") pod \"cluster-samples-operator-7cdcfcf76c-q8vzf\" (UID: \"c2cf5466-b98c-49aa-91b1-eb36d07604e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:52:52.647336 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.647257    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-j86lt"]
Sep 09 23:52:52.650542 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:52:52.650501    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1873ca0_2b52_4016_b708_cde625b04c31.slice/crio-569fa6c33459b865256ed03bd6fadf6f1a53b4a142a055e5d65930875bd24014 WatchSource:0}: Error finding container 569fa6c33459b865256ed03bd6fadf6f1a53b4a142a055e5d65930875bd24014: Status 404 returned error can't find the container with id 569fa6c33459b865256ed03bd6fadf6f1a53b4a142a055e5d65930875bd24014
Sep 09 23:52:52.909331 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.909264    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q8vzf\" (UID: \"c2cf5466-b98c-49aa-91b1-eb36d07604e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:52:52.909331 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:52.909302    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:52.909528 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:52.909402    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 23:52:52.909528 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:52.909436    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 23:52:52.909528 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:52.909463    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls podName:c2cf5466-b98c-49aa-91b1-eb36d07604e1 nodeName:}" failed. No retries permitted until 2026-09-09 23:52:53.909447586 +0000 UTC m=+254.160583612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-q8vzf" (UID: "c2cf5466-b98c-49aa-91b1-eb36d07604e1") : secret "samples-operator-tls" not found
Sep 09 23:52:52.909528 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:52.909485    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls podName:012fd72d-5acd-4896-9a66-ad26dbdc2a4e nodeName:}" failed. No retries permitted until 2026-09-09 23:52:53.909472141 +0000 UTC m=+254.160608163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-m6xqq" (UID: "012fd72d-5acd-4896-9a66-ad26dbdc2a4e") : secret "cluster-monitoring-operator-tls" not found
Sep 09 23:52:53.058654 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:53.058629    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-j86lt" event={"ID":"c1873ca0-2b52-4016-b708-cde625b04c31","Type":"ContainerStarted","Data":"c402c0484f8e6c7376ce18025e74eadb3b5c34e71fb0afe801d5bf9af6f5a1f1"}
Sep 09 23:52:53.059045 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:53.058667    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-j86lt" event={"ID":"c1873ca0-2b52-4016-b708-cde625b04c31","Type":"ContainerStarted","Data":"569fa6c33459b865256ed03bd6fadf6f1a53b4a142a055e5d65930875bd24014"}
Sep 09 23:52:53.076355 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:53.076312    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-j86lt" podStartSLOduration=1.076298542 podStartE2EDuration="1.076298542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:52:53.075171302 +0000 UTC m=+253.326307345" watchObservedRunningTime="2026-09-09 23:52:53.076298542 +0000 UTC m=+253.327434585"
Sep 09 23:52:53.916603 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:53.916566    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q8vzf\" (UID: \"c2cf5466-b98c-49aa-91b1-eb36d07604e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:52:53.916603 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:53.916607    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:53.916831 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:53.916718    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 23:52:53.916831 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:53.916726    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 23:52:53.916831 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:53.916779    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls podName:012fd72d-5acd-4896-9a66-ad26dbdc2a4e nodeName:}" failed. No retries permitted until 2026-09-09 23:52:55.916763149 +0000 UTC m=+256.167899170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-m6xqq" (UID: "012fd72d-5acd-4896-9a66-ad26dbdc2a4e") : secret "cluster-monitoring-operator-tls" not found
Sep 09 23:52:53.916831 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:53.916794    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls podName:c2cf5466-b98c-49aa-91b1-eb36d07604e1 nodeName:}" failed. No retries permitted until 2026-09-09 23:52:55.916787374 +0000 UTC m=+256.167923394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-q8vzf" (UID: "c2cf5466-b98c-49aa-91b1-eb36d07604e1") : secret "samples-operator-tls" not found
Sep 09 23:52:55.930319 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:55.930287    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q8vzf\" (UID: \"c2cf5466-b98c-49aa-91b1-eb36d07604e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:52:55.930319 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:55.930322    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:55.930721 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:55.930435    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 23:52:55.930721 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:55.930493    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls podName:c2cf5466-b98c-49aa-91b1-eb36d07604e1 nodeName:}" failed. No retries permitted until 2026-09-09 23:52:59.930479612 +0000 UTC m=+260.181615638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-q8vzf" (UID: "c2cf5466-b98c-49aa-91b1-eb36d07604e1") : secret "samples-operator-tls" not found
Sep 09 23:52:55.930721 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:55.930435    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 23:52:55.930721 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:55.930568    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls podName:012fd72d-5acd-4896-9a66-ad26dbdc2a4e nodeName:}" failed. No retries permitted until 2026-09-09 23:52:59.930555302 +0000 UTC m=+260.181691326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-m6xqq" (UID: "012fd72d-5acd-4896-9a66-ad26dbdc2a4e") : secret "cluster-monitoring-operator-tls" not found
Sep 09 23:52:56.042517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.042488    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mpflz"]
Sep 09 23:52:56.045635 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.045619    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.047839 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.047815    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 23:52:56.047947 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.047922    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 23:52:56.048035 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.047986    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 23:52:56.048113 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.047989    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 23:52:56.048822 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.048806    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2ktz8\""
Sep 09 23:52:56.056389 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.056369    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mpflz"]
Sep 09 23:52:56.132012 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.131991    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.132124 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.132019    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shmll\" (UniqueName: \"kubernetes.io/projected/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-api-access-shmll\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.132124 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.132044    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-crio-socket\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.132124 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.132112    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.132281 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.132149    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-data-volume\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.233420 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.233392    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.233528 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.233426    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shmll\" (UniqueName: \"kubernetes.io/projected/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-api-access-shmll\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.233528 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.233465    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-crio-socket\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.233528 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.233513    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.233721 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:56.233532    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:56.233721 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.233549    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-data-volume\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.233721 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:56.233594    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls podName:600c0165-b1e7-4a95-bbdb-7f3bcd7764bf nodeName:}" failed. No retries permitted until 2026-09-09 23:52:56.733574974 +0000 UTC m=+256.984710994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpflz" (UID: "600c0165-b1e7-4a95-bbdb-7f3bcd7764bf") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:56.233721 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:56.233631    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 23:52:56.233721 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.233593    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-crio-socket\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.233721 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:56.233714    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm podName:600c0165-b1e7-4a95-bbdb-7f3bcd7764bf nodeName:}" failed. No retries permitted until 2026-09-09 23:52:56.733689781 +0000 UTC m=+256.984825847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpflz" (UID: "600c0165-b1e7-4a95-bbdb-7f3bcd7764bf") : configmap "kube-rbac-proxy" not found
Sep 09 23:52:56.233982 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.233867    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-data-volume\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.243484 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.243461    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shmll\" (UniqueName: \"kubernetes.io/projected/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-api-access-shmll\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.737992 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.737958    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.738206 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.738022    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:56.738206 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:56.738130    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:56.738206 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:56.738190    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 23:52:56.738356 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:56.738201    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls podName:600c0165-b1e7-4a95-bbdb-7f3bcd7764bf nodeName:}" failed. No retries permitted until 2026-09-09 23:52:57.738184268 +0000 UTC m=+257.989320294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpflz" (UID: "600c0165-b1e7-4a95-bbdb-7f3bcd7764bf") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:56.738356 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:56.738274    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm podName:600c0165-b1e7-4a95-bbdb-7f3bcd7764bf nodeName:}" failed. No retries permitted until 2026-09-09 23:52:57.738252553 +0000 UTC m=+257.989388613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpflz" (UID: "600c0165-b1e7-4a95-bbdb-7f3bcd7764bf") : configmap "kube-rbac-proxy" not found
Sep 09 23:52:56.950699 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.950668    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl"]
Sep 09 23:52:56.953730 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.953713    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl"
Sep 09 23:52:56.956843 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.956813    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 09 23:52:56.956946 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.956817    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q8ct2\""
Sep 09 23:52:56.956946 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.956817    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 09 23:52:56.963133 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:56.963111    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl"]
Sep 09 23:52:57.040939 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:57.040881    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9stw\" (UniqueName: \"kubernetes.io/projected/1b6a789d-180e-4cbe-911d-820b513fb316-kube-api-access-m9stw\") pod \"migrator-fdbb8fdff-fgzkl\" (UID: \"1b6a789d-180e-4cbe-911d-820b513fb316\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl"
Sep 09 23:52:57.142105 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:57.142081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9stw\" (UniqueName: \"kubernetes.io/projected/1b6a789d-180e-4cbe-911d-820b513fb316-kube-api-access-m9stw\") pod \"migrator-fdbb8fdff-fgzkl\" (UID: \"1b6a789d-180e-4cbe-911d-820b513fb316\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl"
Sep 09 23:52:57.149664 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:57.149639    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9stw\" (UniqueName: \"kubernetes.io/projected/1b6a789d-180e-4cbe-911d-820b513fb316-kube-api-access-m9stw\") pod \"migrator-fdbb8fdff-fgzkl\" (UID: \"1b6a789d-180e-4cbe-911d-820b513fb316\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl"
Sep 09 23:52:57.263531 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:57.263508    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl"
Sep 09 23:52:57.527741 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:57.527709    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl"]
Sep 09 23:52:57.530424 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:52:57.530397    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6a789d_180e_4cbe_911d_820b513fb316.slice/crio-63e0f3351853eae33ce18ff2eff9410e98eb205592a55b941429fc101208df8b WatchSource:0}: Error finding container 63e0f3351853eae33ce18ff2eff9410e98eb205592a55b941429fc101208df8b: Status 404 returned error can't find the container with id 63e0f3351853eae33ce18ff2eff9410e98eb205592a55b941429fc101208df8b
Sep 09 23:52:57.746364 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:57.746333    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:57.746478 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:57.746430    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:57.746478 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:57.746459    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 23:52:57.746563 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:57.746527    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm podName:600c0165-b1e7-4a95-bbdb-7f3bcd7764bf nodeName:}" failed. No retries permitted until 2026-09-09 23:52:59.746508399 +0000 UTC m=+259.997644420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpflz" (UID: "600c0165-b1e7-4a95-bbdb-7f3bcd7764bf") : configmap "kube-rbac-proxy" not found
Sep 09 23:52:57.746563 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:57.746533    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:57.746644 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:57.746577    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls podName:600c0165-b1e7-4a95-bbdb-7f3bcd7764bf nodeName:}" failed. No retries permitted until 2026-09-09 23:52:59.746564438 +0000 UTC m=+259.997700460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpflz" (UID: "600c0165-b1e7-4a95-bbdb-7f3bcd7764bf") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:58.075319 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:58.075253    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl" event={"ID":"1b6a789d-180e-4cbe-911d-820b513fb316","Type":"ContainerStarted","Data":"63e0f3351853eae33ce18ff2eff9410e98eb205592a55b941429fc101208df8b"}
Sep 09 23:52:58.212216 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:58.212189    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wj527_555a4739-f6bd-4762-9564-7593b7a4f04e/dns-node-resolver/0.log"
Sep 09 23:52:59.012438 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:59.012265    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l72vc_b62491be-c176-4206-9d7b-84caeae4c67d/node-ca/0.log"
Sep 09 23:52:59.079877 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:59.079849    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl" event={"ID":"1b6a789d-180e-4cbe-911d-820b513fb316","Type":"ContainerStarted","Data":"9634e5ced8170738acc56adef2b243a12642f94d29c190e26c94be860a96bfdb"}
Sep 09 23:52:59.080159 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:59.079896    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl" event={"ID":"1b6a789d-180e-4cbe-911d-820b513fb316","Type":"ContainerStarted","Data":"314c80ea8cb621f3af25baad9dba870fbcbe3db589550a5ee5258ad7aa10aad2"}
Sep 09 23:52:59.093783 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:59.093718    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-fgzkl" podStartSLOduration=1.7050358060000002 podStartE2EDuration="3.093702506s" totalImagesPullingTime="1.3886667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:52:59.092872817 +0000 UTC m=+259.344008860" watchObservedRunningTime="2026-09-09 23:52:59.093702506 +0000 UTC m=+259.344838549"
Sep 09 23:52:59.760563 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:59.760533    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:59.760746 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:59.760621    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:52:59.760746 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:59.760663    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 23:52:59.760746 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:59.760725    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:59.760850 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:59.760732    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm podName:600c0165-b1e7-4a95-bbdb-7f3bcd7764bf nodeName:}" failed. No retries permitted until 2026-09-09 23:53:03.760714068 +0000 UTC m=+264.011850089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpflz" (UID: "600c0165-b1e7-4a95-bbdb-7f3bcd7764bf") : configmap "kube-rbac-proxy" not found
Sep 09 23:52:59.760850 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:59.760785    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls podName:600c0165-b1e7-4a95-bbdb-7f3bcd7764bf nodeName:}" failed. No retries permitted until 2026-09-09 23:53:03.760767618 +0000 UTC m=+264.011903645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpflz" (UID: "600c0165-b1e7-4a95-bbdb-7f3bcd7764bf") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:52:59.812296 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:59.812264    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ctkq2_96cde171-d33b-4082-b727-bfd2d2574b7e/serve-healthcheck-canary/0.log"
Sep 09 23:52:59.962133 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:59.962110    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q8vzf\" (UID: \"c2cf5466-b98c-49aa-91b1-eb36d07604e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:52:59.962226 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:52:59.962179    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:52:59.962264 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:59.962244    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 23:52:59.962264 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:59.962262    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 23:52:59.962322 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:59.962299    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls podName:012fd72d-5acd-4896-9a66-ad26dbdc2a4e nodeName:}" failed. No retries permitted until 2026-09-09 23:53:07.962288017 +0000 UTC m=+268.213424038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-m6xqq" (UID: "012fd72d-5acd-4896-9a66-ad26dbdc2a4e") : secret "cluster-monitoring-operator-tls" not found
Sep 09 23:52:59.962322 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:52:59.962311    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls podName:c2cf5466-b98c-49aa-91b1-eb36d07604e1 nodeName:}" failed. No retries permitted until 2026-09-09 23:53:07.962305106 +0000 UTC m=+268.213441128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-q8vzf" (UID: "c2cf5466-b98c-49aa-91b1-eb36d07604e1") : secret "samples-operator-tls" not found
Sep 09 23:53:03.789706 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:03.789605    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:53:03.789706 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:03.789669    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:53:03.790198 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:03.789753    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:53:03.790198 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:03.789821    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls podName:600c0165-b1e7-4a95-bbdb-7f3bcd7764bf nodeName:}" failed. No retries permitted until 2026-09-09 23:53:11.789802949 +0000 UTC m=+272.040938970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpflz" (UID: "600c0165-b1e7-4a95-bbdb-7f3bcd7764bf") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:53:03.790198 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:03.789762    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 23:53:03.790198 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:03.789904    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm podName:600c0165-b1e7-4a95-bbdb-7f3bcd7764bf nodeName:}" failed. No retries permitted until 2026-09-09 23:53:11.789888548 +0000 UTC m=+272.041024569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpflz" (UID: "600c0165-b1e7-4a95-bbdb-7f3bcd7764bf") : configmap "kube-rbac-proxy" not found
Sep 09 23:53:08.022293 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:08.022252    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:53:08.022814 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:08.022409    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 23:53:08.022814 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:08.022451    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q8vzf\" (UID: \"c2cf5466-b98c-49aa-91b1-eb36d07604e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:53:08.022814 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:08.022475    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls podName:012fd72d-5acd-4896-9a66-ad26dbdc2a4e nodeName:}" failed. No retries permitted until 2026-09-09 23:53:24.022457076 +0000 UTC m=+284.273593098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-m6xqq" (UID: "012fd72d-5acd-4896-9a66-ad26dbdc2a4e") : secret "cluster-monitoring-operator-tls" not found
Sep 09 23:53:08.025008 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:08.024983    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2cf5466-b98c-49aa-91b1-eb36d07604e1-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-q8vzf\" (UID: \"c2cf5466-b98c-49aa-91b1-eb36d07604e1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:53:08.088354 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:08.088323    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"
Sep 09 23:53:08.360355 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:08.360286    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf"]
Sep 09 23:53:09.110880 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:09.110849    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf" event={"ID":"c2cf5466-b98c-49aa-91b1-eb36d07604e1","Type":"ContainerStarted","Data":"152800c39ab324fdad322dbeab3946c4924482b9f181d7c01bf0b7358ee0817b"}
Sep 09 23:53:11.119369 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:11.119329    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf" event={"ID":"c2cf5466-b98c-49aa-91b1-eb36d07604e1","Type":"ContainerStarted","Data":"e7b8001fd4dd10e912344bb5f5450245d8e642c83c8e3fab5adc926ae1360a48"}
Sep 09 23:53:11.119799 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:11.119386    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf" event={"ID":"c2cf5466-b98c-49aa-91b1-eb36d07604e1","Type":"ContainerStarted","Data":"5f73a862dd4292e2e85162d1f41b2f0e73692fe0fa2704d44c16f0beeac61769"}
Sep 09 23:53:11.137342 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:11.137290    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-q8vzf" podStartSLOduration=17.258207937 podStartE2EDuration="19.137273605s" totalImagesPullingTime="1.879065668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:11.13634475 +0000 UTC m=+271.387480793" watchObservedRunningTime="2026-09-09 23:53:11.137273605 +0000 UTC m=+271.388409647"
Sep 09 23:53:11.854496 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:11.854460    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:53:11.854676 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:11.854522    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:53:11.854676 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:11.854599    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 23:53:11.854676 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:11.854661    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls podName:600c0165-b1e7-4a95-bbdb-7f3bcd7764bf nodeName:}" failed. No retries permitted until 2026-09-09 23:53:27.854645547 +0000 UTC m=+288.105781571 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpflz" (UID: "600c0165-b1e7-4a95-bbdb-7f3bcd7764bf") : secret "insights-runtime-extractor-tls" not found
Sep 09 23:53:11.854988 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:11.854970    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:53:21.828882 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:21.828826    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-867bcc698d-spnq8" podUID="d59d7cbb-123e-4203-907e-58c96c6ef1f9"
Sep 09 23:53:21.828882 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:21.828857    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zz486" podUID="a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2"
Sep 09 23:53:21.828882 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:21.828826    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw" podUID="6543ed96-cfa4-4666-8547-ec6fd01de0c1"
Sep 09 23:53:22.155624 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:22.155542    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:53:22.155624 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:22.155573    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:53:22.155843 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:22.155768    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zz486"
Sep 09 23:53:24.044154 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:24.044105    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:53:24.046607 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:24.046585    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/012fd72d-5acd-4896-9a66-ad26dbdc2a4e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-m6xqq\" (UID: \"012fd72d-5acd-4896-9a66-ad26dbdc2a4e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:53:24.295136 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:24.295071    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"
Sep 09 23:53:24.562805 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:24.562664    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq"]
Sep 09 23:53:24.565090 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:53:24.565047    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012fd72d_5acd_4896_9a66_ad26dbdc2a4e.slice/crio-5c1231f8c0dcc25edf04826b0821dda4cf8928f7af0446cc0b431d75e10939cb WatchSource:0}: Error finding container 5c1231f8c0dcc25edf04826b0821dda4cf8928f7af0446cc0b431d75e10939cb: Status 404 returned error can't find the container with id 5c1231f8c0dcc25edf04826b0821dda4cf8928f7af0446cc0b431d75e10939cb
Sep 09 23:53:25.054494 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.054468    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:53:25.054851 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.054520    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:53:25.054851 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.054560    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:53:25.057213 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.057181    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"image-registry-867bcc698d-spnq8\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") " pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:53:25.057213 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.057202    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2-metrics-tls\") pod \"dns-default-zz486\" (UID: \"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2\") " pod="openshift-dns/dns-default-zz486"
Sep 09 23:53:25.057360 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.057268    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6543ed96-cfa4-4666-8547-ec6fd01de0c1-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8jvvw\" (UID: \"6543ed96-cfa4-4666-8547-ec6fd01de0c1\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:53:25.158826 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.158797    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mqnps\""
Sep 09 23:53:25.159728 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.159711    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j4rs2\""
Sep 09 23:53:25.159848 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.159825    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rr4p2\""
Sep 09 23:53:25.164976 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.164953    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq" event={"ID":"012fd72d-5acd-4896-9a66-ad26dbdc2a4e","Type":"ContainerStarted","Data":"5c1231f8c0dcc25edf04826b0821dda4cf8928f7af0446cc0b431d75e10939cb"}
Sep 09 23:53:25.167084 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.167065    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:53:25.167084 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.167080    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw"
Sep 09 23:53:25.167196 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.167100    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zz486"
Sep 09 23:53:25.444909 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.444707    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-8jvvw"]
Sep 09 23:53:25.447512 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:53:25.447479    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6543ed96_cfa4_4666_8547_ec6fd01de0c1.slice/crio-10f09d5dbb953177260fa8b688fa420935030d0dae8ef91665a8fc99d3976e17 WatchSource:0}: Error finding container 10f09d5dbb953177260fa8b688fa420935030d0dae8ef91665a8fc99d3976e17: Status 404 returned error can't find the container with id 10f09d5dbb953177260fa8b688fa420935030d0dae8ef91665a8fc99d3976e17
Sep 09 23:53:25.448536 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.448487    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-867bcc698d-spnq8"]
Sep 09 23:53:25.451299 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:25.451079    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zz486"]
Sep 09 23:53:25.454141 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:53:25.454111    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59d7cbb_123e_4203_907e_58c96c6ef1f9.slice/crio-6718375e1e66822aedb5c1837c2fc114c3c95fb7f1595eff2b839b14a5baf40a WatchSource:0}: Error finding container 6718375e1e66822aedb5c1837c2fc114c3c95fb7f1595eff2b839b14a5baf40a: Status 404 returned error can't find the container with id 6718375e1e66822aedb5c1837c2fc114c3c95fb7f1595eff2b839b14a5baf40a
Sep 09 23:53:25.454935 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:53:25.454902    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f6437f_cb3a_45e1_8dd8_6cb24482bcc2.slice/crio-98be2b1e22f9ce15afd8859958a8ef67a52efa86630040a531b9e470a35aff0e WatchSource:0}: Error finding container 98be2b1e22f9ce15afd8859958a8ef67a52efa86630040a531b9e470a35aff0e: Status 404 returned error can't find the container with id 98be2b1e22f9ce15afd8859958a8ef67a52efa86630040a531b9e470a35aff0e
Sep 09 23:53:26.171814 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.171777    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-867bcc698d-spnq8" event={"ID":"d59d7cbb-123e-4203-907e-58c96c6ef1f9","Type":"ContainerStarted","Data":"dea740d2136fee65aed9d1a8b4286ae8d0bd109256d36c1ef5278d9d60f202e8"}
Sep 09 23:53:26.172293 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.171838    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-867bcc698d-spnq8" event={"ID":"d59d7cbb-123e-4203-907e-58c96c6ef1f9","Type":"ContainerStarted","Data":"6718375e1e66822aedb5c1837c2fc114c3c95fb7f1595eff2b839b14a5baf40a"}
Sep 09 23:53:26.172293 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.171865    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:53:26.173107 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.173082    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw" event={"ID":"6543ed96-cfa4-4666-8547-ec6fd01de0c1","Type":"ContainerStarted","Data":"10f09d5dbb953177260fa8b688fa420935030d0dae8ef91665a8fc99d3976e17"}
Sep 09 23:53:26.174239 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.174211    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zz486" event={"ID":"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2","Type":"ContainerStarted","Data":"98be2b1e22f9ce15afd8859958a8ef67a52efa86630040a531b9e470a35aff0e"}
Sep 09 23:53:26.707496 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.707418    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-867bcc698d-spnq8" podStartSLOduration=285.707394223 podStartE2EDuration="4m45.707394223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:26.190424212 +0000 UTC m=+286.441560256" watchObservedRunningTime="2026-09-09 23:53:26.707394223 +0000 UTC m=+286.958530264"
Sep 09 23:53:26.708102 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.708073    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6"]
Sep 09 23:53:26.712771 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.712749    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6"
Sep 09 23:53:26.715694 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.715665    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 23:53:26.715812 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.715751    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bvkvz\""
Sep 09 23:53:26.722005 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.721946    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6"]
Sep 09 23:53:26.769179 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.769147    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6079083-3c7a-4020-ba22-599cf000e8d4-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vs6h6\" (UID: \"e6079083-3c7a-4020-ba22-599cf000e8d4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6"
Sep 09 23:53:26.870574 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:26.870533    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6079083-3c7a-4020-ba22-599cf000e8d4-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vs6h6\" (UID: \"e6079083-3c7a-4020-ba22-599cf000e8d4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6"
Sep 09 23:53:26.870784 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:26.870693    2556 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 09 23:53:26.870784 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:26.870774    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6079083-3c7a-4020-ba22-599cf000e8d4-tls-certificates podName:e6079083-3c7a-4020-ba22-599cf000e8d4 nodeName:}" failed. No retries permitted until 2026-09-09 23:53:27.370753586 +0000 UTC m=+287.621889607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e6079083-3c7a-4020-ba22-599cf000e8d4-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-vs6h6" (UID: "e6079083-3c7a-4020-ba22-599cf000e8d4") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 09 23:53:27.178672 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:27.178634    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq" event={"ID":"012fd72d-5acd-4896-9a66-ad26dbdc2a4e","Type":"ContainerStarted","Data":"23a7944a0fe99113238be7d477f740bf9549f092536199e87daf01423b624026"}
Sep 09 23:53:27.180095 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:27.180068    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw" event={"ID":"6543ed96-cfa4-4666-8547-ec6fd01de0c1","Type":"ContainerStarted","Data":"f8e2a20d402723f35938d6035c7e3e89f5abd4ba3eaab09d7a9cf936efb9e5ed"}
Sep 09 23:53:27.194454 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:27.194406    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-m6xqq" podStartSLOduration=33.529280893 podStartE2EDuration="35.194386817s" totalImagesPullingTime="1.665105924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:27.19251487 +0000 UTC m=+287.443650913" watchObservedRunningTime="2026-09-09 23:53:27.194386817 +0000 UTC m=+287.445522862"
Sep 09 23:53:27.206234 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:27.206191    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-8jvvw" podStartSLOduration=262.646462731 podStartE2EDuration="4m24.206176903s" totalImagesPullingTime="1.559714172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:27.205804169 +0000 UTC m=+287.456940214" watchObservedRunningTime="2026-09-09 23:53:27.206176903 +0000 UTC m=+287.457312947"
Sep 09 23:53:27.375104 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:27.375020    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6079083-3c7a-4020-ba22-599cf000e8d4-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vs6h6\" (UID: \"e6079083-3c7a-4020-ba22-599cf000e8d4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6"
Sep 09 23:53:27.377737 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:27.377707    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e6079083-3c7a-4020-ba22-599cf000e8d4-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vs6h6\" (UID: \"e6079083-3c7a-4020-ba22-599cf000e8d4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6"
Sep 09 23:53:27.624520 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:27.624492    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6"
Sep 09 23:53:27.878292 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:27.878264    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:53:27.880654 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:27.880605    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/600c0165-b1e7-4a95-bbdb-7f3bcd7764bf-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpflz\" (UID: \"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf\") " pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:53:27.892644 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:27.892626    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6"]
Sep 09 23:53:27.894891 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:53:27.894865    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6079083_3c7a_4020_ba22_599cf000e8d4.slice/crio-7b974a60403a2c6f24c765bad953abf51cc896218ceec4aca20c51dc47a1f38b WatchSource:0}: Error finding container 7b974a60403a2c6f24c765bad953abf51cc896218ceec4aca20c51dc47a1f38b: Status 404 returned error can't find the container with id 7b974a60403a2c6f24c765bad953abf51cc896218ceec4aca20c51dc47a1f38b
Sep 09 23:53:28.157026 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:28.156966    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mpflz"
Sep 09 23:53:28.186716 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:28.186686    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zz486" event={"ID":"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2","Type":"ContainerStarted","Data":"a0e646931d0790cee3848780e2d72e406ebb80fb37d6fadb497dbee5b0fb0eaa"}
Sep 09 23:53:28.187127 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:28.186742    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zz486" event={"ID":"a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2","Type":"ContainerStarted","Data":"1a6359a14c4c59237931e09198091dd6aa8af238d57b700440b7f0c5086d2e59"}
Sep 09 23:53:28.187127 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:28.186822    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zz486"
Sep 09 23:53:28.188107 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:28.188078    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6" event={"ID":"e6079083-3c7a-4020-ba22-599cf000e8d4","Type":"ContainerStarted","Data":"7b974a60403a2c6f24c765bad953abf51cc896218ceec4aca20c51dc47a1f38b"}
Sep 09 23:53:28.428006 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:28.427922    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zz486" podStartSLOduration=251.202017014 podStartE2EDuration="4m13.427907954s" totalImagesPullingTime="2.22589094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:28.202367943 +0000 UTC m=+288.453503987" watchObservedRunningTime="2026-09-09 23:53:28.427907954 +0000 UTC m=+288.679044057"
Sep 09 23:53:28.428490 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:28.428475    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mpflz"]
Sep 09 23:53:28.430622 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:53:28.430594    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600c0165_b1e7_4a95_bbdb_7f3bcd7764bf.slice/crio-6226ccf512efba22e6ee6f5484c54474e7591d4771eedcedf9473a1dd8c06c21 WatchSource:0}: Error finding container 6226ccf512efba22e6ee6f5484c54474e7591d4771eedcedf9473a1dd8c06c21: Status 404 returned error can't find the container with id 6226ccf512efba22e6ee6f5484c54474e7591d4771eedcedf9473a1dd8c06c21
Sep 09 23:53:29.193703 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:29.193663    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6" event={"ID":"e6079083-3c7a-4020-ba22-599cf000e8d4","Type":"ContainerStarted","Data":"57a327ef65d1b86ca39fa4790e8dfccf276ed3a028cb2b66fdadfc7e50012f95"}
Sep 09 23:53:29.194150 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:29.194018    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6"
Sep 09 23:53:29.195789 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:29.195751    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpflz" event={"ID":"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf","Type":"ContainerStarted","Data":"4107a451542853f198cee251e10ab65fab379554db759f497674d6fe2e5b52c3"}
Sep 09 23:53:29.195901 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:29.195810    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpflz" event={"ID":"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf","Type":"ContainerStarted","Data":"6226ccf512efba22e6ee6f5484c54474e7591d4771eedcedf9473a1dd8c06c21"}
Sep 09 23:53:29.199776 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:29.199756    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6"
Sep 09 23:53:29.210496 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:29.210456    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vs6h6" podStartSLOduration=2.060181419 podStartE2EDuration="3.210443095s" totalImagesPullingTime="1.150261676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:29.209524452 +0000 UTC m=+289.460660510" watchObservedRunningTime="2026-09-09 23:53:29.210443095 +0000 UTC m=+289.461579186"
Sep 09 23:53:30.201953 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:30.201908    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpflz" event={"ID":"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf","Type":"ContainerStarted","Data":"ece0e999f4d7ca8996924d498aead7284b1315a2775ec9ba16df5c9664364893"}
Sep 09 23:53:31.208424 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:31.208385    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpflz" event={"ID":"600c0165-b1e7-4a95-bbdb-7f3bcd7764bf","Type":"ContainerStarted","Data":"7e8b855369f7ff7bb1efa46344b94c2cb36a3d2e8751fa00b60ce30dee09c10e"}
Sep 09 23:53:31.227417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:31.227359    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mpflz" podStartSLOduration=33.186224887 podStartE2EDuration="35.227343845s" totalImagesPullingTime="2.041118958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:52:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:31.226649641 +0000 UTC m=+291.477785685" watchObservedRunningTime="2026-09-09 23:53:31.227343845 +0000 UTC m=+291.478479888"
Sep 09 23:53:33.264399 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:33.264353    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-plj9c" podUID="21369195-b9f6-4653-ae29-d2a7a07eb1e2"
Sep 09 23:53:35.110922 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.110888    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vlxfk"]
Sep 09 23:53:35.115600 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.115580    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.121015 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.120992    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 23:53:35.121134 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.121016    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kgm5n\""
Sep 09 23:53:35.121134 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.121037    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 23:53:35.121134 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.121044    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 23:53:35.121742 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.121725    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 23:53:35.129089 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.129068    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-root\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.129172 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.129100    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-tls\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.129172 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.129127    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m95tz\" (UniqueName: \"kubernetes.io/projected/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-kube-api-access-m95tz\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.129266 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.129188    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.129266 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.129238    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-textfile\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.129326 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.129310    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-metrics-client-ca\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.129360 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.129336    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-wtmp\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.129360 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.129354    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-sys\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.129426 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.129371    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.229922 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.229891    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-root\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.229922 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.229921    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-tls\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230079 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.229938    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95tz\" (UniqueName: \"kubernetes.io/projected/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-kube-api-access-m95tz\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230079 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.229993    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-root\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230079 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:35.230018    2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 23:53:35.230079 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.230049    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230214 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:35.230084    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-tls podName:8389ea18-0b63-4ccf-b5ec-2e24b99b35fe nodeName:}" failed. No retries permitted until 2026-09-09 23:53:35.730070198 +0000 UTC m=+295.981206232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-tls") pod "node-exporter-vlxfk" (UID: "8389ea18-0b63-4ccf-b5ec-2e24b99b35fe") : secret "node-exporter-tls" not found
Sep 09 23:53:35.230214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.230107    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-textfile\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.230141    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-metrics-client-ca\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.230165    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-wtmp\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.230190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-sys\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230420 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.230214    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230420 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.230260    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-sys\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230420 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.230335    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-wtmp\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230515 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.230442    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-textfile\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230716 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.230694    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-metrics-client-ca\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.230771 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.230703    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.232458 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.232436    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.237886 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.237866    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95tz\" (UniqueName: \"kubernetes.io/projected/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-kube-api-access-m95tz\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.735039 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.735006    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-tls\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:35.737736 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:35.737711    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8389ea18-0b63-4ccf-b5ec-2e24b99b35fe-node-exporter-tls\") pod \"node-exporter-vlxfk\" (UID: \"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe\") " pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:36.024103 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.024028    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vlxfk"
Sep 09 23:53:36.039550 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:53:36.039522    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8389ea18_0b63_4ccf_b5ec_2e24b99b35fe.slice/crio-fae24bdba6786fdd79d1cf8f20ce470aaf6861f8590144dc69fb42e3068107b7 WatchSource:0}: Error finding container fae24bdba6786fdd79d1cf8f20ce470aaf6861f8590144dc69fb42e3068107b7: Status 404 returned error can't find the container with id fae24bdba6786fdd79d1cf8f20ce470aaf6861f8590144dc69fb42e3068107b7
Sep 09 23:53:36.152393 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.152363    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 23:53:36.157191 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.157171    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.160155 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.160127    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tkfs8\""
Sep 09 23:53:36.160263 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.160166    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 23:53:36.160263 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.160226    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 23:53:36.160263 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.160240    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 23:53:36.160508 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.160495    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 23:53:36.160996 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.160980    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 23:53:36.161095 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.161016    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 23:53:36.161159 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.161107    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 23:53:36.161203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.161156    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 23:53:36.161241 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.161218    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 23:53:36.173165 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.173142    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 23:53:36.228456 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.228436    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vlxfk" event={"ID":"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe","Type":"ContainerStarted","Data":"fae24bdba6786fdd79d1cf8f20ce470aaf6861f8590144dc69fb42e3068107b7"}
Sep 09 23:53:36.238760 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.238742    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.238824 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.238767    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.238824 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.238790    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-volume\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.238824 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.238806    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.238943 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.238845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.238943 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.238877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.238943 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.238897    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.238943 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.238912    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ntk\" (UniqueName: \"kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-kube-api-access-29ntk\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.239083 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.238946    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.239083 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.238976    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-out\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.239083 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.239009    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.239083 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.239025    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.239083 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.239081    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-web-config\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.339835 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.339777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.339835 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.339815    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-out\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.339967 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.339839    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340023 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.339969    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340100 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.340020    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-web-config\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340100 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.340081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340211 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:36.340122    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-trusted-ca-bundle podName:82e67a34-21b4-443f-b98d-6b3734ee8eca nodeName:}" failed. No retries permitted until 2026-09-09 23:53:36.840101818 +0000 UTC m=+297.091237853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca") : configmap references non-existent config key: ca-bundle.crt
Sep 09 23:53:36.340211 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.340139    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340211 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.340182    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-volume\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340211 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.340206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340414 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.340237    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340414 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.340273    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340414 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.340298    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340414 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.340322    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ntk\" (UniqueName: \"kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-kube-api-access-29ntk\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340605 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:36.340548    2556 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 23:53:36.340605 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.340559    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.340704 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:36.340607    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls podName:82e67a34-21b4-443f-b98d-6b3734ee8eca nodeName:}" failed. No retries permitted until 2026-09-09 23:53:36.840590474 +0000 UTC m=+297.091726507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca") : secret "alertmanager-main-tls" not found
Sep 09 23:53:36.340903 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.340873    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.342986 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.342956    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.343309 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.343290    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-out\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.343452 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.343414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.343578 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.343561    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.343642 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.343567    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-volume\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.344035 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.344018    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.344112 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.344046    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-web-config\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.344734 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.344715    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.348940 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.348908    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ntk\" (UniqueName: \"kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-kube-api-access-29ntk\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.843887 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.843850    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.844048 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.843924    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:36.844171 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:36.844068    2556 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 23:53:36.844171 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:53:36.844130    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls podName:82e67a34-21b4-443f-b98d-6b3734ee8eca nodeName:}" failed. No retries permitted until 2026-09-09 23:53:37.844112377 +0000 UTC m=+298.095248403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca") : secret "alertmanager-main-tls" not found
Sep 09 23:53:36.844815 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:36.844775    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:37.233433 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:37.233402    2556 generic.go:356] "Generic (PLEG): container finished" podID="8389ea18-0b63-4ccf-b5ec-2e24b99b35fe" containerID="c27b40f039cb1cbbe4d560f1fada9977ae9755b7bd744a1cfb4f64c792db70c9" exitCode=0
Sep 09 23:53:37.233779 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:37.233438    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vlxfk" event={"ID":"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe","Type":"ContainerDied","Data":"c27b40f039cb1cbbe4d560f1fada9977ae9755b7bd744a1cfb4f64c792db70c9"}
Sep 09 23:53:37.852974 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:37.852933    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:37.855394 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:37.855366    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:37.966538 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:37.966511    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:53:38.080138 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.080113    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"]
Sep 09 23:53:38.083771 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.083756    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.086148 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.086117    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 09 23:53:38.086245 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.086190    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-356ukccqgush3\""
Sep 09 23:53:38.086322 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.086246    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 09 23:53:38.086322 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.086262    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6fnq8\""
Sep 09 23:53:38.086322 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.086273    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 09 23:53:38.086470 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.086275    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 09 23:53:38.087181 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.087163    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 09 23:53:38.094809 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.094788    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"]
Sep 09 23:53:38.155012 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.154960    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e19cad74-ef52-400e-b684-b37fbbc770e0-metrics-client-ca\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.155012 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.155003    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.155154 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.155027    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7jw9\" (UniqueName: \"kubernetes.io/projected/e19cad74-ef52-400e-b684-b37fbbc770e0-kube-api-access-m7jw9\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.155154 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.155110    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.155154 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.155129    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.155249 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.155163    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-tls\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.155249 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.155198    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.155249 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.155227    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-grpc-tls\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.197966 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.197948    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zz486"
Sep 09 23:53:38.239392 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.239370    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vlxfk" event={"ID":"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe","Type":"ContainerStarted","Data":"47638d0d8d599ae9b3d1ffee7107197c2656050696d1aa5b493112436f587972"}
Sep 09 23:53:38.239691 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.239409    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vlxfk" event={"ID":"8389ea18-0b63-4ccf-b5ec-2e24b99b35fe","Type":"ContainerStarted","Data":"1e3cb380281cc49c1acc60063b15afda29bd6e2aba702fdb6acaa68d66935215"}
Sep 09 23:53:38.252228 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.252206    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 23:53:38.254271 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:53:38.254248    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e67a34_21b4_443f_b98d_6b3734ee8eca.slice/crio-c8221ac1418b7daa6f2e3b01a5464332673fd23a603edf8c52f7de31bac67396 WatchSource:0}: Error finding container c8221ac1418b7daa6f2e3b01a5464332673fd23a603edf8c52f7de31bac67396: Status 404 returned error can't find the container with id c8221ac1418b7daa6f2e3b01a5464332673fd23a603edf8c52f7de31bac67396
Sep 09 23:53:38.255682 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.255661    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-tls\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.255782 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.255727    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.255782 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.255760    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-grpc-tls\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.255890 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.255825    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e19cad74-ef52-400e-b684-b37fbbc770e0-metrics-client-ca\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.256006 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.255897    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.256006 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.255934    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7jw9\" (UniqueName: \"kubernetes.io/projected/e19cad74-ef52-400e-b684-b37fbbc770e0-kube-api-access-m7jw9\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.256006 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.255980    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.256168 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.256009    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.256631 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.256602    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e19cad74-ef52-400e-b684-b37fbbc770e0-metrics-client-ca\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.259497 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.259226    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-tls\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.259497 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.259230    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.259497 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.259457    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-grpc-tls\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.259703 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.259533    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.260174 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.260149    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.260260 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.260237    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e19cad74-ef52-400e-b684-b37fbbc770e0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.272667 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.272648    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7jw9\" (UniqueName: \"kubernetes.io/projected/e19cad74-ef52-400e-b684-b37fbbc770e0-kube-api-access-m7jw9\") pod \"thanos-querier-74cd8cddb-tmzdc\" (UID: \"e19cad74-ef52-400e-b684-b37fbbc770e0\") " pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.282653 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.282609    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vlxfk" podStartSLOduration=2.220140417 podStartE2EDuration="3.282595819s" totalImagesPullingTime="1.003277476s" totalInitContainerRuntime="59.177926ms" isStatefulPod=true podCreationTimestamp="2026-09-09 23:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:38.281439928 +0000 UTC m=+298.532575972" watchObservedRunningTime="2026-09-09 23:53:38.282595819 +0000 UTC m=+298.533731862"
Sep 09 23:53:38.393297 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.393276    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:38.664365 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:38.664343    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"]
Sep 09 23:53:38.666600 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:53:38.666569    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19cad74_ef52_400e_b684_b37fbbc770e0.slice/crio-54c0744b453f9b22f184d748444d475ae52bbd33de82d9fb115e3e0382337cc8 WatchSource:0}: Error finding container 54c0744b453f9b22f184d748444d475ae52bbd33de82d9fb115e3e0382337cc8: Status 404 returned error can't find the container with id 54c0744b453f9b22f184d748444d475ae52bbd33de82d9fb115e3e0382337cc8
Sep 09 23:53:39.245729 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:39.245666    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc" event={"ID":"e19cad74-ef52-400e-b684-b37fbbc770e0","Type":"ContainerStarted","Data":"54c0744b453f9b22f184d748444d475ae52bbd33de82d9fb115e3e0382337cc8"}
Sep 09 23:53:39.247548 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:39.247437    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerStarted","Data":"c8221ac1418b7daa6f2e3b01a5464332673fd23a603edf8c52f7de31bac67396"}
Sep 09 23:53:40.253565 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:40.253395    2556 generic.go:356] "Generic (PLEG): container finished" podID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerID="c4c93c461d1b746f31219545922b2a9217076505e2c6b45e7f0c6f84e206687a" exitCode=0
Sep 09 23:53:40.253565 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:40.253461    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerDied","Data":"c4c93c461d1b746f31219545922b2a9217076505e2c6b45e7f0c6f84e206687a"}
Sep 09 23:53:40.472412 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:40.472387    2556 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 23:53:41.265040 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.264997    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc" event={"ID":"e19cad74-ef52-400e-b684-b37fbbc770e0","Type":"ContainerStarted","Data":"aaec7baf553079a3e8bb604d00506b007456c154ef4024f6f7308055082f9424"}
Sep 09 23:53:41.265502 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.265099    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc" event={"ID":"e19cad74-ef52-400e-b684-b37fbbc770e0","Type":"ContainerStarted","Data":"45ea7b2f6fd544c433ab80842ef296607fafecf21183a03687ff8d227f98ecd6"}
Sep 09 23:53:41.265502 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.265128    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc" event={"ID":"e19cad74-ef52-400e-b684-b37fbbc770e0","Type":"ContainerStarted","Data":"ae3ecffcbd8ebf5f3a6d93ebe7fcd93d86b07f956be401f213795ad4d758ae24"}
Sep 09 23:53:41.294916 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.294882    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 23:53:41.299070 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.299038    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.301557 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.301534    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 23:53:41.301729 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.301577    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 23:53:41.301899 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.301808    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 23:53:41.302098 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.302049    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 23:53:41.302235 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.302216    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 23:53:41.302346 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.302306    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 23:53:41.302346 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.302334    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-chvp8\""
Sep 09 23:53:41.302841 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.302819    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 23:53:41.303151 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.303000    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 23:53:41.303278 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.303256    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 23:53:41.303353 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.303287    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 23:53:41.303405 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.303367    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 23:53:41.304100 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.303914    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 23:53:41.304775 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.304581    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 23:53:41.304775 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.304740    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-alfl05itv9dh2\""
Sep 09 23:53:41.304973 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.304901    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 23:53:41.305250 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.305124    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 23:53:41.314883 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.314855    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 23:53:41.383969 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.383943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384111 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.383976    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384111 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.383995    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384111 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384022    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384111 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384093    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384318 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384127    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384318 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384166    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384318 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384194    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384318 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384217    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384318 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384242    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384318 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384267    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384318 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384289    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384318 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384315    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384655 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384339    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384655 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384354    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4ll8\" (UniqueName: \"kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-kube-api-access-q4ll8\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384655 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384402    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384655 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384426    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384655 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384441    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384655 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384455    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.384655 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.384534    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.485891 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.485865    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4ll8\" (UniqueName: \"kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-kube-api-access-q4ll8\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486030 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.485908    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486030 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.485930    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486030 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.485953    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486030 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.485976    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486030 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486027    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486082    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486109    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486138    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486200    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486222    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486253    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486282    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486303    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486343 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486325    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486821 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486350    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486821 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486373    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486821 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486402    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486821 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486443    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.486821 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486797    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.487091 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486831    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.488390 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.488329    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.489207 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.486802    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.489207 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.489184    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.489489 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.489424    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.489610 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.489586    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.489988 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.489969    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.490159 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.490132    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.490387 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.490364    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-web-config\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.490464 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.490394    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.490950 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.490927    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.491685 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.491201    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config-out\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.491685 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.491642    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.491842 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.491735    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.491842 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.491761    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.491842 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.491769    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.492451 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.492419    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.492735 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.492718    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.493690 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.493670    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4ll8\" (UniqueName: \"kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-kube-api-access-q4ll8\") pod \"prometheus-k8s-0\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:41.611618 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:41.611530    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:42.273483 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:42.273444    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc" event={"ID":"e19cad74-ef52-400e-b684-b37fbbc770e0","Type":"ContainerStarted","Data":"5ec0e486d613e5a1fe94b1875af9d9039ec7cecd971e48fafa1cc815e30eaf4c"}
Sep 09 23:53:42.274209 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:42.274186    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc" event={"ID":"e19cad74-ef52-400e-b684-b37fbbc770e0","Type":"ContainerStarted","Data":"80a7ce9d6afa82ab5104b05e208e7ce2f28426e1d8157f2a1dda24335de0d382"}
Sep 09 23:53:42.274387 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:42.274372    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc" event={"ID":"e19cad74-ef52-400e-b684-b37fbbc770e0","Type":"ContainerStarted","Data":"b14aecfef2de5e32e3937c1e79ecf3a90d2200f0ed12be055beec03635394305"}
Sep 09 23:53:42.274529 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:42.274515    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:42.276502 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:42.276483    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerStarted","Data":"3c6fa3d557068784ce3bd40167aa4409e00678de42b5027145c922646bb7a48a"}
Sep 09 23:53:42.276598 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:42.276531    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerStarted","Data":"32b0e223c52229da3a5b36e62b49bd429050fca0b349863174c50145e3d894e6"}
Sep 09 23:53:42.276598 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:42.276558    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerStarted","Data":"eab28446962398f7d861594f5031c0385fa583607e7311e4a85b0e834f5df306"}
Sep 09 23:53:42.294491 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:42.294449    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc" podStartSLOduration=1.095636673 podStartE2EDuration="4.294432199s" totalImagesPullingTime="3.198795526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:53:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:42.292603785 +0000 UTC m=+302.543739828" watchObservedRunningTime="2026-09-09 23:53:42.294432199 +0000 UTC m=+302.545568241"
Sep 09 23:53:42.324538 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:42.324515    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 23:53:42.326881 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:53:42.326857    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4194cf36_5d6c_4311_8f9b_3e02787a4f1a.slice/crio-405f7540598bfd3fa7593c23a3f57de3ed77d0acec59932fea83ea269abc1210 WatchSource:0}: Error finding container 405f7540598bfd3fa7593c23a3f57de3ed77d0acec59932fea83ea269abc1210: Status 404 returned error can't find the container with id 405f7540598bfd3fa7593c23a3f57de3ed77d0acec59932fea83ea269abc1210
Sep 09 23:53:43.283251 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:43.283216    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerStarted","Data":"1dfd21163feb7c1321f2161689bc3976dd615169ead8c945032f17716b0275cd"}
Sep 09 23:53:43.283756 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:43.283323    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerStarted","Data":"450b5a37d214669e9282dc4f5838418961d7d8fff4b2adfe556a5a11e2bea3b6"}
Sep 09 23:53:43.283756 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:43.283367    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerStarted","Data":"325d0ec0ade136306f16652647df82c399e0efd5bd13d57c8648d7f6c499e495"}
Sep 09 23:53:43.284912 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:43.284883    2556 generic.go:356] "Generic (PLEG): container finished" podID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerID="790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af" exitCode=0
Sep 09 23:53:43.285020 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:43.284914    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerDied","Data":"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"}
Sep 09 23:53:43.285020 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:43.284970    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerStarted","Data":"405f7540598bfd3fa7593c23a3f57de3ed77d0acec59932fea83ea269abc1210"}
Sep 09 23:53:43.286043 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:43.286028    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 23:53:43.308794 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:43.308755    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.280213044 podStartE2EDuration="7.308739807s" totalImagesPullingTime="3.012260096s" totalInitContainerRuntime="16.266667ms" isStatefulPod=false podCreationTimestamp="2026-09-09 23:53:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:43.30788717 +0000 UTC m=+303.559023225" watchObservedRunningTime="2026-09-09 23:53:43.308739807 +0000 UTC m=+303.559875851"
Sep 09 23:53:44.213211 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:44.213180    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-867bcc698d-spnq8"]
Sep 09 23:53:44.218359 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:44.218326    2556 patch_prober.go:28] interesting pod/image-registry-867bcc698d-spnq8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 23:53:44.218524 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:44.218382    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-867bcc698d-spnq8" podUID="d59d7cbb-123e-4203-907e-58c96c6ef1f9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 23:53:45.262470 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:45.262434    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:53:47.306209 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:47.306166    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerStarted","Data":"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"}
Sep 09 23:53:47.306694 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:47.306253    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerStarted","Data":"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"}
Sep 09 23:53:47.306694 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:47.306299    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerStarted","Data":"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"}
Sep 09 23:53:47.306694 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:47.306342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerStarted","Data":"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"}
Sep 09 23:53:47.306694 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:47.306385    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerStarted","Data":"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"}
Sep 09 23:53:47.306694 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:47.306436    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerStarted","Data":"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"}
Sep 09 23:53:47.336157 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:47.336098    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.310561873 podStartE2EDuration="6.336077531s" totalImagesPullingTime="3.002807685s" totalInitContainerRuntime="22.707973ms" isStatefulPod=false podCreationTimestamp="2026-09-09 23:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:53:47.334658533 +0000 UTC m=+307.585794604" watchObservedRunningTime="2026-09-09 23:53:47.336077531 +0000 UTC m=+307.587213569"
Sep 09 23:53:49.293119 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:49.293090    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74cd8cddb-tmzdc"
Sep 09 23:53:51.611911 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:51.611867    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:53:54.217981 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:53:54.217946    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:54:09.236987 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.236912    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-867bcc698d-spnq8" podUID="d59d7cbb-123e-4203-907e-58c96c6ef1f9" containerName="registry" containerID="cri-o://dea740d2136fee65aed9d1a8b4286ae8d0bd109256d36c1ef5278d9d60f202e8" gracePeriod=30
Sep 09 23:54:09.392530 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.392481    2556 generic.go:356] "Generic (PLEG): container finished" podID="d59d7cbb-123e-4203-907e-58c96c6ef1f9" containerID="dea740d2136fee65aed9d1a8b4286ae8d0bd109256d36c1ef5278d9d60f202e8" exitCode=0
Sep 09 23:54:09.392667 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.392552    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-867bcc698d-spnq8" event={"ID":"d59d7cbb-123e-4203-907e-58c96c6ef1f9","Type":"ContainerDied","Data":"dea740d2136fee65aed9d1a8b4286ae8d0bd109256d36c1ef5278d9d60f202e8"}
Sep 09 23:54:09.485436 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.485411    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:54:09.610708 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.610646    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-trusted-ca\") pod \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") "
Sep 09 23:54:09.610708 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.610681    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-installation-pull-secrets\") pod \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") "
Sep 09 23:54:09.610708 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.610701    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-image-registry-private-configuration\") pod \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") "
Sep 09 23:54:09.610926 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.610725    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d59d7cbb-123e-4203-907e-58c96c6ef1f9-ca-trust-extracted\") pod \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") "
Sep 09 23:54:09.610926 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.610770    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-bound-sa-token\") pod \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") "
Sep 09 23:54:09.610926 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.610799    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52jm5\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-kube-api-access-52jm5\") pod \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") "
Sep 09 23:54:09.610926 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.610842    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") pod \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") "
Sep 09 23:54:09.610926 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.610889    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-certificates\") pod \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\" (UID: \"d59d7cbb-123e-4203-907e-58c96c6ef1f9\") "
Sep 09 23:54:09.611196 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.611167    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d59d7cbb-123e-4203-907e-58c96c6ef1f9" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:54:09.611589 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.611564    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d59d7cbb-123e-4203-907e-58c96c6ef1f9" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:54:09.613438 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.613391    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d59d7cbb-123e-4203-907e-58c96c6ef1f9" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:54:09.613581 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.613525    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d59d7cbb-123e-4203-907e-58c96c6ef1f9" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:54:09.613581 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.613532    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-kube-api-access-52jm5" (OuterVolumeSpecName: "kube-api-access-52jm5") pod "d59d7cbb-123e-4203-907e-58c96c6ef1f9" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9"). InnerVolumeSpecName "kube-api-access-52jm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:54:09.613718 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.613610    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d59d7cbb-123e-4203-907e-58c96c6ef1f9" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:54:09.613718 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.613625    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d59d7cbb-123e-4203-907e-58c96c6ef1f9" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:54:09.619348 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.619323    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59d7cbb-123e-4203-907e-58c96c6ef1f9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d59d7cbb-123e-4203-907e-58c96c6ef1f9" (UID: "d59d7cbb-123e-4203-907e-58c96c6ef1f9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:54:09.712166 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.712148    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-trusted-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:09.712166 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.712167    2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-installation-pull-secrets\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:09.712285 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.712177    2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d59d7cbb-123e-4203-907e-58c96c6ef1f9-image-registry-private-configuration\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:09.712285 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.712186    2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d59d7cbb-123e-4203-907e-58c96c6ef1f9-ca-trust-extracted\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:09.712285 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.712194    2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-bound-sa-token\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:09.712285 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.712202    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52jm5\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-kube-api-access-52jm5\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:09.712285 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.712210    2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-tls\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:09.712285 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:09.712219    2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d59d7cbb-123e-4203-907e-58c96c6ef1f9-registry-certificates\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:10.397404 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:10.397371    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-867bcc698d-spnq8" event={"ID":"d59d7cbb-123e-4203-907e-58c96c6ef1f9","Type":"ContainerDied","Data":"6718375e1e66822aedb5c1837c2fc114c3c95fb7f1595eff2b839b14a5baf40a"}
Sep 09 23:54:10.397404 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:10.397402    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-867bcc698d-spnq8"
Sep 09 23:54:10.397849 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:10.397428    2556 scope.go:117] "RemoveContainer" containerID="dea740d2136fee65aed9d1a8b4286ae8d0bd109256d36c1ef5278d9d60f202e8"
Sep 09 23:54:10.415531 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:10.415510    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-867bcc698d-spnq8"]
Sep 09 23:54:10.423373 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:10.423340    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-867bcc698d-spnq8"]
Sep 09 23:54:12.267294 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:12.267262    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d59d7cbb-123e-4203-907e-58c96c6ef1f9" path="/var/lib/kubelet/pods/d59d7cbb-123e-4203-907e-58c96c6ef1f9/volumes"
Sep 09 23:54:41.611985 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:41.611957    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:54:41.629599 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:41.629575    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:54:42.545815 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:42.545773    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:54:54.049555 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:54.049523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:54:54.051832 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:54.051810    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21369195-b9f6-4653-ae29-d2a7a07eb1e2-metrics-certs\") pod \"network-metrics-daemon-plj9c\" (UID: \"21369195-b9f6-4653-ae29-d2a7a07eb1e2\") " pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:54:54.266657 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:54.266627    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lqch4\""
Sep 09 23:54:54.273621 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:54.273600    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plj9c"
Sep 09 23:54:54.554181 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:54.554159    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-plj9c"]
Sep 09 23:54:54.556866 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:54:54.556841    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21369195_b9f6_4653_ae29_d2a7a07eb1e2.slice/crio-0cac1f30aa9e973484cea2873911b32c7b198face105a6ee6fcf3f233cf5b588 WatchSource:0}: Error finding container 0cac1f30aa9e973484cea2873911b32c7b198face105a6ee6fcf3f233cf5b588: Status 404 returned error can't find the container with id 0cac1f30aa9e973484cea2873911b32c7b198face105a6ee6fcf3f233cf5b588
Sep 09 23:54:54.578215 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:54.578183    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plj9c" event={"ID":"21369195-b9f6-4653-ae29-d2a7a07eb1e2","Type":"ContainerStarted","Data":"0cac1f30aa9e973484cea2873911b32c7b198face105a6ee6fcf3f233cf5b588"}
Sep 09 23:54:55.448209 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.448175    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 23:54:55.448601 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.448531    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="alertmanager" containerID="cri-o://eab28446962398f7d861594f5031c0385fa583607e7311e4a85b0e834f5df306" gracePeriod=120
Sep 09 23:54:55.448601 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.448579    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy-metric" containerID="cri-o://450b5a37d214669e9282dc4f5838418961d7d8fff4b2adfe556a5a11e2bea3b6" gracePeriod=120
Sep 09 23:54:55.448714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.448594    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy-web" containerID="cri-o://3c6fa3d557068784ce3bd40167aa4409e00678de42b5027145c922646bb7a48a" gracePeriod=120
Sep 09 23:54:55.448714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.448646    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="prom-label-proxy" containerID="cri-o://1dfd21163feb7c1321f2161689bc3976dd615169ead8c945032f17716b0275cd" gracePeriod=120
Sep 09 23:54:55.448714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.448663    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="config-reloader" containerID="cri-o://32b0e223c52229da3a5b36e62b49bd429050fca0b349863174c50145e3d894e6" gracePeriod=120
Sep 09 23:54:55.448714 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.448613    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy" containerID="cri-o://325d0ec0ade136306f16652647df82c399e0efd5bd13d57c8648d7f6c499e495" gracePeriod=120
Sep 09 23:54:55.586514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.586485    2556 generic.go:356] "Generic (PLEG): container finished" podID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerID="1dfd21163feb7c1321f2161689bc3976dd615169ead8c945032f17716b0275cd" exitCode=0
Sep 09 23:54:55.586592 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.586516    2556 generic.go:356] "Generic (PLEG): container finished" podID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerID="325d0ec0ade136306f16652647df82c399e0efd5bd13d57c8648d7f6c499e495" exitCode=0
Sep 09 23:54:55.586592 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.586529    2556 generic.go:356] "Generic (PLEG): container finished" podID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerID="32b0e223c52229da3a5b36e62b49bd429050fca0b349863174c50145e3d894e6" exitCode=0
Sep 09 23:54:55.586592 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.586540    2556 generic.go:356] "Generic (PLEG): container finished" podID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerID="eab28446962398f7d861594f5031c0385fa583607e7311e4a85b0e834f5df306" exitCode=0
Sep 09 23:54:55.586592 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.586550    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerDied","Data":"1dfd21163feb7c1321f2161689bc3976dd615169ead8c945032f17716b0275cd"}
Sep 09 23:54:55.586739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.586618    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerDied","Data":"325d0ec0ade136306f16652647df82c399e0efd5bd13d57c8648d7f6c499e495"}
Sep 09 23:54:55.586739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.586645    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerDied","Data":"32b0e223c52229da3a5b36e62b49bd429050fca0b349863174c50145e3d894e6"}
Sep 09 23:54:55.586739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:55.586669    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerDied","Data":"eab28446962398f7d861594f5031c0385fa583607e7311e4a85b0e834f5df306"}
Sep 09 23:54:56.592965 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.592588    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plj9c" event={"ID":"21369195-b9f6-4653-ae29-d2a7a07eb1e2","Type":"ContainerStarted","Data":"0a0eb8392adeb4883ae976b3096dbf13414b627cc74cc382a14afa04486ed540"}
Sep 09 23:54:56.592965 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.592652    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plj9c" event={"ID":"21369195-b9f6-4653-ae29-d2a7a07eb1e2","Type":"ContainerStarted","Data":"30b297ea84c1e25d1e140e60a5c11414c5312f5acd4c9a7811b95fceb1a86c0c"}
Sep 09 23:54:56.597348 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.597323    2556 generic.go:356] "Generic (PLEG): container finished" podID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerID="450b5a37d214669e9282dc4f5838418961d7d8fff4b2adfe556a5a11e2bea3b6" exitCode=0
Sep 09 23:54:56.597348 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.597347    2556 generic.go:356] "Generic (PLEG): container finished" podID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerID="3c6fa3d557068784ce3bd40167aa4409e00678de42b5027145c922646bb7a48a" exitCode=0
Sep 09 23:54:56.597480 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.597386    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerDied","Data":"450b5a37d214669e9282dc4f5838418961d7d8fff4b2adfe556a5a11e2bea3b6"}
Sep 09 23:54:56.597480 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.597439    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerDied","Data":"3c6fa3d557068784ce3bd40167aa4409e00678de42b5027145c922646bb7a48a"}
Sep 09 23:54:56.609196 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.609154    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-plj9c" podStartSLOduration=374.58330895 podStartE2EDuration="6m15.609140068s" totalImagesPullingTime="1.025831118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:54:56.608615237 +0000 UTC m=+376.859751293" watchObservedRunningTime="2026-09-09 23:54:56.609140068 +0000 UTC m=+376.860276111"
Sep 09 23:54:56.701623 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.701602    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:56.775374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.775305    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-web\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.775374 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.775344    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-trusted-ca-bundle\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.775531 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.775393    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-tls-assets\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.775531 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.775427    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-web-config\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.775531 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.775458    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.775531 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.775482    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-volume\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.775531 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.775514    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-out\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.775784 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.775549    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29ntk\" (UniqueName: \"kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-kube-api-access-29ntk\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.775846 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.775809    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:54:56.775958 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.775939    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-metrics-client-ca\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.776030 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.775991    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-main-db\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.776030 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.776019    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.776157 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.776066    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.776157 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.776115    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-cluster-tls-config\") pod \"82e67a34-21b4-443f-b98d-6b3734ee8eca\" (UID: \"82e67a34-21b4-443f-b98d-6b3734ee8eca\") "
Sep 09 23:54:56.776401 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.776382    2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.777580 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.777478    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:54:56.777931 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.777905    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:54:56.778553 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.778421    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:54:56.778553 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.778467    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:54:56.778553 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.778523    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:54:56.779884 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.779846    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-out" (OuterVolumeSpecName: "config-out") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:54:56.779976 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.779875    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-kube-api-access-29ntk" (OuterVolumeSpecName: "kube-api-access-29ntk") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "kube-api-access-29ntk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:54:56.780112 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.780090    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:54:56.780241 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.780218    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-volume" (OuterVolumeSpecName: "config-volume") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:54:56.780335 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.780315    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:54:56.784464 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.784441    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:54:56.791302 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.791277    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-web-config" (OuterVolumeSpecName: "web-config") pod "82e67a34-21b4-443f-b98d-6b3734ee8eca" (UID: "82e67a34-21b4-443f-b98d-6b3734ee8eca"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:54:56.876719 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876696    2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-tls-assets\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.876719 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876716    2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-web-config\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.876829 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876726    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.876829 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876735    2556 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.876829 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876744    2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-config-out\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.876829 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876751    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ntk\" (UniqueName: \"kubernetes.io/projected/82e67a34-21b4-443f-b98d-6b3734ee8eca-kube-api-access-29ntk\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.876829 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876760    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e67a34-21b4-443f-b98d-6b3734ee8eca-metrics-client-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.876829 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876768    2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/82e67a34-21b4-443f-b98d-6b3734ee8eca-alertmanager-main-db\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.876829 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876777    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.876829 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876787    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-main-tls\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.876829 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876795    2556 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-cluster-tls-config\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:56.876829 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:56.876804    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82e67a34-21b4-443f-b98d-6b3734ee8eca-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:54:57.605216 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.605188    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.605569 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.605190    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"82e67a34-21b4-443f-b98d-6b3734ee8eca","Type":"ContainerDied","Data":"c8221ac1418b7daa6f2e3b01a5464332673fd23a603edf8c52f7de31bac67396"}
Sep 09 23:54:57.605569 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.605292    2556 scope.go:117] "RemoveContainer" containerID="1dfd21163feb7c1321f2161689bc3976dd615169ead8c945032f17716b0275cd"
Sep 09 23:54:57.613186 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.613165    2556 scope.go:117] "RemoveContainer" containerID="450b5a37d214669e9282dc4f5838418961d7d8fff4b2adfe556a5a11e2bea3b6"
Sep 09 23:54:57.624016 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.623995    2556 scope.go:117] "RemoveContainer" containerID="325d0ec0ade136306f16652647df82c399e0efd5bd13d57c8648d7f6c499e495"
Sep 09 23:54:57.631318 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.631293    2556 scope.go:117] "RemoveContainer" containerID="3c6fa3d557068784ce3bd40167aa4409e00678de42b5027145c922646bb7a48a"
Sep 09 23:54:57.633955 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.633934    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 23:54:57.641970 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.641952    2556 scope.go:117] "RemoveContainer" containerID="32b0e223c52229da3a5b36e62b49bd429050fca0b349863174c50145e3d894e6"
Sep 09 23:54:57.642794 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.642772    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 23:54:57.648803 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.648788    2556 scope.go:117] "RemoveContainer" containerID="eab28446962398f7d861594f5031c0385fa583607e7311e4a85b0e834f5df306"
Sep 09 23:54:57.655862 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.655845    2556 scope.go:117] "RemoveContainer" containerID="c4c93c461d1b746f31219545922b2a9217076505e2c6b45e7f0c6f84e206687a"
Sep 09 23:54:57.682679 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.682657    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 23:54:57.682947 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.682935    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="alertmanager"
Sep 09 23:54:57.682988 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.682949    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="alertmanager"
Sep 09 23:54:57.682988 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.682970    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59d7cbb-123e-4203-907e-58c96c6ef1f9" containerName="registry"
Sep 09 23:54:57.682988 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.682976    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59d7cbb-123e-4203-907e-58c96c6ef1f9" containerName="registry"
Sep 09 23:54:57.682988 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.682982    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy-web"
Sep 09 23:54:57.682988 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.682987    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy-web"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.682995    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy-metric"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683001    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy-metric"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683008    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="config-reloader"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683012    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="config-reloader"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683018    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683023    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683029    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="prom-label-proxy"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683033    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="prom-label-proxy"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683039    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="init-config-reloader"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683044    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="init-config-reloader"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683098    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d59d7cbb-123e-4203-907e-58c96c6ef1f9" containerName="registry"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683106    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="alertmanager"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683111    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy-metric"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683117    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="config-reloader"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683123    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="prom-label-proxy"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683128    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy-web"
Sep 09 23:54:57.683140 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.683135    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" containerName="kube-rbac-proxy"
Sep 09 23:54:57.687528 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.687513    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.693127 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.693106    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 23:54:57.693214 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.693109    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 23:54:57.693448 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.693376    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 23:54:57.693562 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.693505    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 23:54:57.693562 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.693521    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 23:54:57.693665 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.693560    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 23:54:57.693665 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.693626    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tkfs8\""
Sep 09 23:54:57.693818 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.693799    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 23:54:57.694347 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.694330    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 23:54:57.699677 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.699659    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 23:54:57.705885 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.705863    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 23:54:57.783704 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.783678    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.783798 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.783711    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/624a9187-3dcc-419b-a663-91159b9879ef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.783798 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.783773    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/624a9187-3dcc-419b-a663-91159b9879ef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.783885 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.783805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg4zz\" (UniqueName: \"kubernetes.io/projected/624a9187-3dcc-419b-a663-91159b9879ef-kube-api-access-zg4zz\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.783885 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.783831    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/624a9187-3dcc-419b-a663-91159b9879ef-config-out\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.783885 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.783855    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-config-volume\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.783885 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.783870    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.784017 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.783887    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.784017 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.783901    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-web-config\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.784017 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.783964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/624a9187-3dcc-419b-a663-91159b9879ef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.784017 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.783983    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.784017 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.784003    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.784198 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.784025    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/624a9187-3dcc-419b-a663-91159b9879ef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884398 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884342    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-config-volume\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884398 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884371    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884398 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884389    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884600 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884404    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-web-config\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884600 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884437    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/624a9187-3dcc-419b-a663-91159b9879ef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884600 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884459    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884600 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884489    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884600 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884516    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/624a9187-3dcc-419b-a663-91159b9879ef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884600 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884553    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884600 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884577    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/624a9187-3dcc-419b-a663-91159b9879ef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884928 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884624    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/624a9187-3dcc-419b-a663-91159b9879ef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884928 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884650    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4zz\" (UniqueName: \"kubernetes.io/projected/624a9187-3dcc-419b-a663-91159b9879ef-kube-api-access-zg4zz\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.884928 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.884676    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/624a9187-3dcc-419b-a663-91159b9879ef-config-out\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.885651 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.885626    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/624a9187-3dcc-419b-a663-91159b9879ef-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.885738 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.885665    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/624a9187-3dcc-419b-a663-91159b9879ef-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.886551 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.886519    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/624a9187-3dcc-419b-a663-91159b9879ef-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.888490 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.888325    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.888490 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.888402    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.888490 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.888408    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.888767 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.888750    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-web-config\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.888767 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.888758    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.888880 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.888827    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-config-volume\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.889122 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.889091    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/624a9187-3dcc-419b-a663-91159b9879ef-tls-assets\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.889462 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.889439    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/624a9187-3dcc-419b-a663-91159b9879ef-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.889549 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.889471    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/624a9187-3dcc-419b-a663-91159b9879ef-config-out\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.894617 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.894596    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4zz\" (UniqueName: \"kubernetes.io/projected/624a9187-3dcc-419b-a663-91159b9879ef-kube-api-access-zg4zz\") pod \"alertmanager-main-0\" (UID: \"624a9187-3dcc-419b-a663-91159b9879ef\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:57.996857 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:57.996837    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 23:54:58.267377 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:58.267346    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82e67a34-21b4-443f-b98d-6b3734ee8eca" path="/var/lib/kubelet/pods/82e67a34-21b4-443f-b98d-6b3734ee8eca/volumes"
Sep 09 23:54:58.274495 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:58.274470    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 23:54:58.276656 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:54:58.276633    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624a9187_3dcc_419b_a663_91159b9879ef.slice/crio-51bdf1ec16fde9578b47a1c9b7fc731bc4c4d7bd51d26cfa092d36dc55334ed5 WatchSource:0}: Error finding container 51bdf1ec16fde9578b47a1c9b7fc731bc4c4d7bd51d26cfa092d36dc55334ed5: Status 404 returned error can't find the container with id 51bdf1ec16fde9578b47a1c9b7fc731bc4c4d7bd51d26cfa092d36dc55334ed5
Sep 09 23:54:58.611294 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:58.611220    2556 generic.go:356] "Generic (PLEG): container finished" podID="624a9187-3dcc-419b-a663-91159b9879ef" containerID="bc738bd56a5ecbc8ee80755444d8dd2ef2f05404c4d68b90533c07221f1997b1" exitCode=0
Sep 09 23:54:58.611671 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:58.611304    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"624a9187-3dcc-419b-a663-91159b9879ef","Type":"ContainerDied","Data":"bc738bd56a5ecbc8ee80755444d8dd2ef2f05404c4d68b90533c07221f1997b1"}
Sep 09 23:54:58.611671 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:58.611377    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"624a9187-3dcc-419b-a663-91159b9879ef","Type":"ContainerStarted","Data":"51bdf1ec16fde9578b47a1c9b7fc731bc4c4d7bd51d26cfa092d36dc55334ed5"}
Sep 09 23:54:59.619782 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.619751    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"624a9187-3dcc-419b-a663-91159b9879ef","Type":"ContainerStarted","Data":"1b91395ad6bde8832f7df31a0d67d6b2ce35a5f04d654a55aa60e342f6d610b0"}
Sep 09 23:54:59.620117 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.619838    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"624a9187-3dcc-419b-a663-91159b9879ef","Type":"ContainerStarted","Data":"066c2eb4db5de87f921b0fdcf1c1788a308325a09e2cf3ef492aa89f8def3f0d"}
Sep 09 23:54:59.620117 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.619867    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"624a9187-3dcc-419b-a663-91159b9879ef","Type":"ContainerStarted","Data":"1e9a7025eca1bd66e0ebb011df904af5ed2db500b8f28eae2765afd5823ffbf8"}
Sep 09 23:54:59.620117 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.619895    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"624a9187-3dcc-419b-a663-91159b9879ef","Type":"ContainerStarted","Data":"d797c5897215cb14b916325bdeba03e3947c53cee847b6e04b9ec1351302890d"}
Sep 09 23:54:59.620117 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.619918    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"624a9187-3dcc-419b-a663-91159b9879ef","Type":"ContainerStarted","Data":"182e234823866d3e965192e4d703221bac84494a7bd92d523d4e42eee5871515"}
Sep 09 23:54:59.620117 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.619940    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"624a9187-3dcc-419b-a663-91159b9879ef","Type":"ContainerStarted","Data":"79d734f0048f6f9ea45fdde01d6e93c41b80ba278fcdc12509dce8aeea59686d"}
Sep 09 23:54:59.653405 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.653355    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.636475972 podStartE2EDuration="2.653338606s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.862634ms" isStatefulPod=false podCreationTimestamp="2026-09-09 23:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:54:59.651543401 +0000 UTC m=+379.902679445" watchObservedRunningTime="2026-09-09 23:54:59.653338606 +0000 UTC m=+379.904474650"
Sep 09 23:54:59.732644 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.732618    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 23:54:59.732920 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.732892    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="prometheus" containerID="cri-o://504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04" gracePeriod=600
Sep 09 23:54:59.733023 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.732926    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="thanos-sidecar" containerID="cri-o://4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe" gracePeriod=600
Sep 09 23:54:59.733023 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.732937    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="config-reloader" containerID="cri-o://87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0" gracePeriod=600
Sep 09 23:54:59.733023 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.732956    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy-web" containerID="cri-o://a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100" gracePeriod=600
Sep 09 23:54:59.733023 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.732898    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy" containerID="cri-o://d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3" gracePeriod=600
Sep 09 23:54:59.733235 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.732935    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69" gracePeriod=600
Sep 09 23:54:59.974567 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:54:59.974539    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.103142 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103112    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-serving-certs-ca-bundle\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103142 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103148    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config-out\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103354 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103164    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-tls-assets\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103354 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103184    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4ll8\" (UniqueName: \"kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-kube-api-access-q4ll8\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103354 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103201    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-1\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103354 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103241    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-db\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103354 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103271    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-metrics-client-ca\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103354 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103301    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-metrics-client-certs\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103354 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103326    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-thanos-prometheus-http-client-file\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103354 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103350    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-kubelet-serving-ca-bundle\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103382    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-trusted-ca-bundle\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103420    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-grpc-tls\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103467    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-kube-rbac-proxy\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103509    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-0\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103538    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103566    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103595    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103632    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-2\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103643    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:55:00.103739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103675    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-tls\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.103739 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103698    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-web-config\") pod \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\" (UID: \"4194cf36-5d6c-4311-8f9b-3e02787a4f1a\") "
Sep 09 23:55:00.104256 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.103948    2556 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.105166 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.104449    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:55:00.105166 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.104777    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:55:00.105166 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.104842    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:55:00.105402 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.105365    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:55:00.105977 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.105807    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:55:00.106517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.106391    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:55:00.106517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.106422    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 23:55:00.106517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.106486    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:55:00.106517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.106490    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-kube-api-access-q4ll8" (OuterVolumeSpecName: "kube-api-access-q4ll8") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "kube-api-access-q4ll8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:55:00.106517 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.106494    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 23:55:00.106842 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.106780    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:55:00.106842 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.106801    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config-out" (OuterVolumeSpecName: "config-out") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 23:55:00.107425 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.107396    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:55:00.107584 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.107562    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:55:00.107970 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.107947    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:55:00.108693 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.108675    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:55:00.108796 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.108778    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config" (OuterVolumeSpecName: "config") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:55:00.109099 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.109083    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:55:00.117596 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.117576    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-web-config" (OuterVolumeSpecName: "web-config") pod "4194cf36-5d6c-4311-8f9b-3e02787a4f1a" (UID: "4194cf36-5d6c-4311-8f9b-3e02787a4f1a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 23:55:00.204285 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204268    2556 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-kube-rbac-proxy\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204370 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204287    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204370 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204297    2556 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204370 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204307    2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204370 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204316    2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204370 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204325    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204370 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204334    2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204370 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204343    2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-web-config\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204370 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204351    2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-config-out\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204370 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204358    2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-tls-assets\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204370 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204365    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4ll8\" (UniqueName: \"kubernetes.io/projected/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-kube-api-access-q4ll8\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204658 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204373    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204658 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204383    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-k8s-db\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204658 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204391    2556 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-metrics-client-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204658 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204399    2556 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-metrics-client-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204658 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204408    2556 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204658 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204417    2556 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204658 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204425    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.204658 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.204433    2556 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4194cf36-5d6c-4311-8f9b-3e02787a4f1a-secret-grpc-tls\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 09 23:55:00.626686 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.626651    2556 generic.go:356] "Generic (PLEG): container finished" podID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerID="b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69" exitCode=0
Sep 09 23:55:00.626686 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.626682    2556 generic.go:356] "Generic (PLEG): container finished" podID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerID="d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3" exitCode=0
Sep 09 23:55:00.626686 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.626690    2556 generic.go:356] "Generic (PLEG): container finished" podID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerID="a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100" exitCode=0
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.626698    2556 generic.go:356] "Generic (PLEG): container finished" podID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerID="4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe" exitCode=0
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.626706    2556 generic.go:356] "Generic (PLEG): container finished" podID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerID="87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0" exitCode=0
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.626713    2556 generic.go:356] "Generic (PLEG): container finished" podID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerID="504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04" exitCode=0
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.626748    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.626739    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerDied","Data":"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"}
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.626909    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerDied","Data":"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"}
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.626958    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerDied","Data":"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"}
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.626985    2556 scope.go:117] "RemoveContainer" containerID="b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.627006    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerDied","Data":"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"}
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.627050    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerDied","Data":"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"}
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.627109    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerDied","Data":"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"}
Sep 09 23:55:00.627178 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.627147    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4194cf36-5d6c-4311-8f9b-3e02787a4f1a","Type":"ContainerDied","Data":"405f7540598bfd3fa7593c23a3f57de3ed77d0acec59932fea83ea269abc1210"}
Sep 09 23:55:00.635822 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.635802    2556 scope.go:117] "RemoveContainer" containerID="d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"
Sep 09 23:55:00.642927 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.642909    2556 scope.go:117] "RemoveContainer" containerID="a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"
Sep 09 23:55:00.649905 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.649886    2556 scope.go:117] "RemoveContainer" containerID="4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"
Sep 09 23:55:00.651550 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.651531    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 23:55:00.657148 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.657128    2556 scope.go:117] "RemoveContainer" containerID="87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"
Sep 09 23:55:00.657924 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.657902    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 23:55:00.664028 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.664007    2556 scope.go:117] "RemoveContainer" containerID="504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"
Sep 09 23:55:00.670886 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.670867    2556 scope.go:117] "RemoveContainer" containerID="790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"
Sep 09 23:55:00.677262 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.677244    2556 scope.go:117] "RemoveContainer" containerID="b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"
Sep 09 23:55:00.677527 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:55:00.677508    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": container with ID starting with b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69 not found: ID does not exist" containerID="b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"
Sep 09 23:55:00.677624 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.677533    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"} err="failed to get container status \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": rpc error: code = NotFound desc = could not find container \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": container with ID starting with b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69 not found: ID does not exist"
Sep 09 23:55:00.677624 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.677563    2556 scope.go:117] "RemoveContainer" containerID="d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"
Sep 09 23:55:00.677843 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:55:00.677800    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": container with ID starting with d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3 not found: ID does not exist" containerID="d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"
Sep 09 23:55:00.677843 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.677827    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"} err="failed to get container status \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": rpc error: code = NotFound desc = could not find container \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": container with ID starting with d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3 not found: ID does not exist"
Sep 09 23:55:00.677927 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.677846    2556 scope.go:117] "RemoveContainer" containerID="a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"
Sep 09 23:55:00.678232 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:55:00.678205    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": container with ID starting with a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100 not found: ID does not exist" containerID="a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"
Sep 09 23:55:00.678335 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.678237    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"} err="failed to get container status \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": rpc error: code = NotFound desc = could not find container \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": container with ID starting with a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100 not found: ID does not exist"
Sep 09 23:55:00.678335 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.678256    2556 scope.go:117] "RemoveContainer" containerID="4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"
Sep 09 23:55:00.678564 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:55:00.678545    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": container with ID starting with 4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe not found: ID does not exist" containerID="4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"
Sep 09 23:55:00.678601 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.678568    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"} err="failed to get container status \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": rpc error: code = NotFound desc = could not find container \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": container with ID starting with 4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe not found: ID does not exist"
Sep 09 23:55:00.678601 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.678583    2556 scope.go:117] "RemoveContainer" containerID="87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"
Sep 09 23:55:00.678801 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:55:00.678784    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": container with ID starting with 87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0 not found: ID does not exist" containerID="87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"
Sep 09 23:55:00.678833 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.678806    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"} err="failed to get container status \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": rpc error: code = NotFound desc = could not find container \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": container with ID starting with 87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0 not found: ID does not exist"
Sep 09 23:55:00.678833 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.678818    2556 scope.go:117] "RemoveContainer" containerID="504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"
Sep 09 23:55:00.679083 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:55:00.679038    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": container with ID starting with 504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04 not found: ID does not exist" containerID="504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"
Sep 09 23:55:00.679173 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.679088    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"} err="failed to get container status \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": rpc error: code = NotFound desc = could not find container \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": container with ID starting with 504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04 not found: ID does not exist"
Sep 09 23:55:00.679173 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.679108    2556 scope.go:117] "RemoveContainer" containerID="790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"
Sep 09 23:55:00.679425 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:55:00.679376    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": container with ID starting with 790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af not found: ID does not exist" containerID="790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"
Sep 09 23:55:00.679425 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.679404    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"} err="failed to get container status \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": rpc error: code = NotFound desc = could not find container \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": container with ID starting with 790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af not found: ID does not exist"
Sep 09 23:55:00.679539 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.679425    2556 scope.go:117] "RemoveContainer" containerID="b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"
Sep 09 23:55:00.679665 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.679648    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"} err="failed to get container status \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": rpc error: code = NotFound desc = could not find container \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": container with ID starting with b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69 not found: ID does not exist"
Sep 09 23:55:00.679707 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.679664    2556 scope.go:117] "RemoveContainer" containerID="d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"
Sep 09 23:55:00.679876 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.679857    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"} err="failed to get container status \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": rpc error: code = NotFound desc = could not find container \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": container with ID starting with d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3 not found: ID does not exist"
Sep 09 23:55:00.679949 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.679877    2556 scope.go:117] "RemoveContainer" containerID="a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"
Sep 09 23:55:00.680114 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.680095    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"} err="failed to get container status \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": rpc error: code = NotFound desc = could not find container \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": container with ID starting with a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100 not found: ID does not exist"
Sep 09 23:55:00.680159 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.680115    2556 scope.go:117] "RemoveContainer" containerID="4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"
Sep 09 23:55:00.680330 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.680311    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"} err="failed to get container status \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": rpc error: code = NotFound desc = could not find container \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": container with ID starting with 4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe not found: ID does not exist"
Sep 09 23:55:00.680398 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.680332    2556 scope.go:117] "RemoveContainer" containerID="87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"
Sep 09 23:55:00.680551 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.680535    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"} err="failed to get container status \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": rpc error: code = NotFound desc = could not find container \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": container with ID starting with 87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0 not found: ID does not exist"
Sep 09 23:55:00.680618 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.680552    2556 scope.go:117] "RemoveContainer" containerID="504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"
Sep 09 23:55:00.680820 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.680800    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"} err="failed to get container status \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": rpc error: code = NotFound desc = could not find container \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": container with ID starting with 504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04 not found: ID does not exist"
Sep 09 23:55:00.680869 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.680822    2556 scope.go:117] "RemoveContainer" containerID="790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"
Sep 09 23:55:00.681050 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.681031    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"} err="failed to get container status \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": rpc error: code = NotFound desc = could not find container \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": container with ID starting with 790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af not found: ID does not exist"
Sep 09 23:55:00.681127 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.681073    2556 scope.go:117] "RemoveContainer" containerID="b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"
Sep 09 23:55:00.681325 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.681307    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"} err="failed to get container status \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": rpc error: code = NotFound desc = could not find container \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": container with ID starting with b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69 not found: ID does not exist"
Sep 09 23:55:00.681378 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.681326    2556 scope.go:117] "RemoveContainer" containerID="d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"
Sep 09 23:55:00.681539 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.681521    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"} err="failed to get container status \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": rpc error: code = NotFound desc = could not find container \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": container with ID starting with d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3 not found: ID does not exist"
Sep 09 23:55:00.681605 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.681540    2556 scope.go:117] "RemoveContainer" containerID="a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"
Sep 09 23:55:00.681777 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.681756    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"} err="failed to get container status \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": rpc error: code = NotFound desc = could not find container \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": container with ID starting with a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100 not found: ID does not exist"
Sep 09 23:55:00.681823 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.681778    2556 scope.go:117] "RemoveContainer" containerID="4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"
Sep 09 23:55:00.681994 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.681978    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"} err="failed to get container status \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": rpc error: code = NotFound desc = could not find container \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": container with ID starting with 4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe not found: ID does not exist"
Sep 09 23:55:00.682070 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.681995    2556 scope.go:117] "RemoveContainer" containerID="87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"
Sep 09 23:55:00.682237 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.682221    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"} err="failed to get container status \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": rpc error: code = NotFound desc = could not find container \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": container with ID starting with 87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0 not found: ID does not exist"
Sep 09 23:55:00.682305 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.682238    2556 scope.go:117] "RemoveContainer" containerID="504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"
Sep 09 23:55:00.682466 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.682450    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"} err="failed to get container status \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": rpc error: code = NotFound desc = could not find container \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": container with ID starting with 504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04 not found: ID does not exist"
Sep 09 23:55:00.682511 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.682465    2556 scope.go:117] "RemoveContainer" containerID="790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"
Sep 09 23:55:00.682666 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.682648    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"} err="failed to get container status \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": rpc error: code = NotFound desc = could not find container \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": container with ID starting with 790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af not found: ID does not exist"
Sep 09 23:55:00.682726 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.682667    2556 scope.go:117] "RemoveContainer" containerID="b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"
Sep 09 23:55:00.682881 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.682866    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"} err="failed to get container status \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": rpc error: code = NotFound desc = could not find container \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": container with ID starting with b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69 not found: ID does not exist"
Sep 09 23:55:00.682922 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.682880    2556 scope.go:117] "RemoveContainer" containerID="d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"
Sep 09 23:55:00.683065 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.683038    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"} err="failed to get container status \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": rpc error: code = NotFound desc = could not find container \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": container with ID starting with d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3 not found: ID does not exist"
Sep 09 23:55:00.683137 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.683079    2556 scope.go:117] "RemoveContainer" containerID="a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"
Sep 09 23:55:00.683299 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.683280    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"} err="failed to get container status \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": rpc error: code = NotFound desc = could not find container \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": container with ID starting with a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100 not found: ID does not exist"
Sep 09 23:55:00.683341 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.683299    2556 scope.go:117] "RemoveContainer" containerID="4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"
Sep 09 23:55:00.683489 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.683463    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"} err="failed to get container status \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": rpc error: code = NotFound desc = could not find container \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": container with ID starting with 4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe not found: ID does not exist"
Sep 09 23:55:00.683555 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.683490    2556 scope.go:117] "RemoveContainer" containerID="87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"
Sep 09 23:55:00.683766 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.683747    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"} err="failed to get container status \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": rpc error: code = NotFound desc = could not find container \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": container with ID starting with 87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0 not found: ID does not exist"
Sep 09 23:55:00.683813 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.683768    2556 scope.go:117] "RemoveContainer" containerID="504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"
Sep 09 23:55:00.683972 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.683954    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"} err="failed to get container status \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": rpc error: code = NotFound desc = could not find container \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": container with ID starting with 504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04 not found: ID does not exist"
Sep 09 23:55:00.684031 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.683973    2556 scope.go:117] "RemoveContainer" containerID="790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"
Sep 09 23:55:00.684202 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.684178    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"} err="failed to get container status \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": rpc error: code = NotFound desc = could not find container \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": container with ID starting with 790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af not found: ID does not exist"
Sep 09 23:55:00.684253 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.684202    2556 scope.go:117] "RemoveContainer" containerID="b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"
Sep 09 23:55:00.684399 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.684379    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"} err="failed to get container status \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": rpc error: code = NotFound desc = could not find container \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": container with ID starting with b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69 not found: ID does not exist"
Sep 09 23:55:00.684465 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.684400    2556 scope.go:117] "RemoveContainer" containerID="d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"
Sep 09 23:55:00.684610 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.684593    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"} err="failed to get container status \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": rpc error: code = NotFound desc = could not find container \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": container with ID starting with d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3 not found: ID does not exist"
Sep 09 23:55:00.684655 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.684610    2556 scope.go:117] "RemoveContainer" containerID="a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"
Sep 09 23:55:00.684807 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.684789    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"} err="failed to get container status \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": rpc error: code = NotFound desc = could not find container \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": container with ID starting with a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100 not found: ID does not exist"
Sep 09 23:55:00.684844 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.684808    2556 scope.go:117] "RemoveContainer" containerID="4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"
Sep 09 23:55:00.685016 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.685000    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"} err="failed to get container status \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": rpc error: code = NotFound desc = could not find container \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": container with ID starting with 4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe not found: ID does not exist"
Sep 09 23:55:00.685016 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.685015    2556 scope.go:117] "RemoveContainer" containerID="87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"
Sep 09 23:55:00.685224 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.685206    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"} err="failed to get container status \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": rpc error: code = NotFound desc = could not find container \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": container with ID starting with 87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0 not found: ID does not exist"
Sep 09 23:55:00.685263 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.685225    2556 scope.go:117] "RemoveContainer" containerID="504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"
Sep 09 23:55:00.685465 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.685450    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"} err="failed to get container status \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": rpc error: code = NotFound desc = could not find container \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": container with ID starting with 504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04 not found: ID does not exist"
Sep 09 23:55:00.685502 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.685464    2556 scope.go:117] "RemoveContainer" containerID="790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"
Sep 09 23:55:00.685681 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.685663    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"} err="failed to get container status \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": rpc error: code = NotFound desc = could not find container \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": container with ID starting with 790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af not found: ID does not exist"
Sep 09 23:55:00.685743 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.685683    2556 scope.go:117] "RemoveContainer" containerID="b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"
Sep 09 23:55:00.685902 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.685886    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69"} err="failed to get container status \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": rpc error: code = NotFound desc = could not find container \"b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69\": container with ID starting with b1a689d721b71f9f7d6e495cca952cc1d9fc9c5337302d428990b2db5cc65f69 not found: ID does not exist"
Sep 09 23:55:00.685944 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.685902    2556 scope.go:117] "RemoveContainer" containerID="d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"
Sep 09 23:55:00.686152 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.686129    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3"} err="failed to get container status \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": rpc error: code = NotFound desc = could not find container \"d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3\": container with ID starting with d0fbdea11d0c983522837a8b48005b75d1959936c0b8d3ec70ea52e125a0caf3 not found: ID does not exist"
Sep 09 23:55:00.686152 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.686149    2556 scope.go:117] "RemoveContainer" containerID="a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"
Sep 09 23:55:00.686806 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.686506    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100"} err="failed to get container status \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": rpc error: code = NotFound desc = could not find container \"a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100\": container with ID starting with a8eecdae2594eeaa2cd36cb7c6f419acbaaaba7a64bb22e9b82fbb295ba80100 not found: ID does not exist"
Sep 09 23:55:00.686806 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.686531    2556 scope.go:117] "RemoveContainer" containerID="4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"
Sep 09 23:55:00.686961 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.686912    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe"} err="failed to get container status \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": rpc error: code = NotFound desc = could not find container \"4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe\": container with ID starting with 4511e4d71342a53179d66c71a687b93a55548efde1dcc5dcb6b5b67474274afe not found: ID does not exist"
Sep 09 23:55:00.686961 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.686933    2556 scope.go:117] "RemoveContainer" containerID="87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"
Sep 09 23:55:00.687246 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.687207    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0"} err="failed to get container status \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": rpc error: code = NotFound desc = could not find container \"87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0\": container with ID starting with 87fed215db239feb315c84a573addd9d18afe6c38e83916d52dc370c6962e9f0 not found: ID does not exist"
Sep 09 23:55:00.687246 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.687233    2556 scope.go:117] "RemoveContainer" containerID="504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"
Sep 09 23:55:00.687661 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.687509    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04"} err="failed to get container status \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": rpc error: code = NotFound desc = could not find container \"504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04\": container with ID starting with 504f9eea0ca354cb009d796fcf024c6c5c52eba7b3ce756e24835a70d285ed04 not found: ID does not exist"
Sep 09 23:55:00.687661 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.687533    2556 scope.go:117] "RemoveContainer" containerID="790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"
Sep 09 23:55:00.688087 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688047    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af"} err="failed to get container status \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": rpc error: code = NotFound desc = could not find container \"790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af\": container with ID starting with 790822b63e5f62e342b1915aa093f01772877bac7e02525c7f231995501dc1af not found: ID does not exist"
Sep 09 23:55:00.688479 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688460    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 23:55:00.688773 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688758    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="config-reloader"
Sep 09 23:55:00.688847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688776    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="config-reloader"
Sep 09 23:55:00.688847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688789    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy"
Sep 09 23:55:00.688847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688798    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy"
Sep 09 23:55:00.688847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688816    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy-thanos"
Sep 09 23:55:00.688847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688825    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy-thanos"
Sep 09 23:55:00.688847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688837    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="thanos-sidecar"
Sep 09 23:55:00.688847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688845    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="thanos-sidecar"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688856    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="init-config-reloader"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688865    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="init-config-reloader"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688878    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="prometheus"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688887    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="prometheus"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688901    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy-web"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688910    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy-web"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.688988    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="prometheus"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.689000    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="config-reloader"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.689011    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="thanos-sidecar"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.689021    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy-thanos"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.689029    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy"
Sep 09 23:55:00.689203 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.689038    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" containerName="kube-rbac-proxy-web"
Sep 09 23:55:00.694232 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.694213    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.697511 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.697492    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 23:55:00.697586 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.697563    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 23:55:00.697803 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.697784    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 23:55:00.697873 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.697826    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 23:55:00.698030 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.698014    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 23:55:00.698380 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.698366    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 23:55:00.698555 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.698541    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 23:55:00.698629 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.698572    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 23:55:00.698629 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.698587    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 23:55:00.698731 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.698577    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 23:55:00.698950 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.698932    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 23:55:00.699040 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.698954    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 23:55:00.699040 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.698967    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 23:55:00.699040 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.698955    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-chvp8\""
Sep 09 23:55:00.704993 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.702091    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-alfl05itv9dh2\""
Sep 09 23:55:00.706631 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.706603    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 23:55:00.708853 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.708828    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 23:55:00.711499 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.711479    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 23:55:00.810919 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.810864    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.810919 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.810889    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.810919 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.810908    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811083 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.810927    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811083 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.810987    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-config\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811083 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811016    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811083 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811036    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811083 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811068    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3e6c34a8-c8da-4057-9e5d-c8877401f083-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811233 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811088    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811233 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811110    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811233 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811132    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811233 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811149    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811233 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811199    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811233 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811226    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811401 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811244    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3e6c34a8-c8da-4057-9e5d-c8877401f083-config-out\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811401 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811268    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gng8\" (UniqueName: \"kubernetes.io/projected/3e6c34a8-c8da-4057-9e5d-c8877401f083-kube-api-access-7gng8\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811401 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811287    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811401 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811302    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811401 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811330    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.811401 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.811350    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-web-config\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.911744 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.911717    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.911847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.911751    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.911847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.911776    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3e6c34a8-c8da-4057-9e5d-c8877401f083-config-out\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.911847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.911797    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gng8\" (UniqueName: \"kubernetes.io/projected/3e6c34a8-c8da-4057-9e5d-c8877401f083-kube-api-access-7gng8\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.911847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.911827    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912042 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.911851    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912042 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.911879    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912042 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.911904    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-web-config\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912042 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.911951    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912042 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.911975    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912042 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.912004    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912042 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.912035    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.912096    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-config\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.912124    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.912152    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.912178    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3e6c34a8-c8da-4057-9e5d-c8877401f083-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.912206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.912231    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.912258    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.912417 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.912286    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.913013 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.912912    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.913765 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.913736    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.913883 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.913856    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.914368 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.914339    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.914451 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.914422    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.914519 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.914502    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.915035 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.915008    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.915784 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.915758    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3e6c34a8-c8da-4057-9e5d-c8877401f083-config-out\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.916159 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.916134    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.916892 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.916239    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.916892 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.916765    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-config\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.916892 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.916765    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.917551 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.917525    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-web-config\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.917639 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.917620    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.917785 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.917763    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.917999 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.917976    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3e6c34a8-c8da-4057-9e5d-c8877401f083-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.918092 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.917980    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.918092 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.917985    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3e6c34a8-c8da-4057-9e5d-c8877401f083-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.918514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.918488    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3e6c34a8-c8da-4057-9e5d-c8877401f083-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:00.920616 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:00.920593    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gng8\" (UniqueName: \"kubernetes.io/projected/3e6c34a8-c8da-4057-9e5d-c8877401f083-kube-api-access-7gng8\") pod \"prometheus-k8s-0\" (UID: \"3e6c34a8-c8da-4057-9e5d-c8877401f083\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:01.008669 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:01.008643    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:55:01.311617 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:01.311586    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 23:55:01.313764 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:55:01.313732    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6c34a8_c8da_4057_9e5d_c8877401f083.slice/crio-309090b841469013b2de712b8d57959d08e2d93cadaac37ca1f88eec04d6d21b WatchSource:0}: Error finding container 309090b841469013b2de712b8d57959d08e2d93cadaac37ca1f88eec04d6d21b: Status 404 returned error can't find the container with id 309090b841469013b2de712b8d57959d08e2d93cadaac37ca1f88eec04d6d21b
Sep 09 23:55:01.632460 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:01.632382    2556 generic.go:356] "Generic (PLEG): container finished" podID="3e6c34a8-c8da-4057-9e5d-c8877401f083" containerID="382e3920b84fd0b715a03f65a6577561086c000487e12b34d29d8be35f90f707" exitCode=0
Sep 09 23:55:01.632813 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:01.632459    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e6c34a8-c8da-4057-9e5d-c8877401f083","Type":"ContainerDied","Data":"382e3920b84fd0b715a03f65a6577561086c000487e12b34d29d8be35f90f707"}
Sep 09 23:55:01.632813 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:01.632521    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e6c34a8-c8da-4057-9e5d-c8877401f083","Type":"ContainerStarted","Data":"309090b841469013b2de712b8d57959d08e2d93cadaac37ca1f88eec04d6d21b"}
Sep 09 23:55:02.268388 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:02.268354    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4194cf36-5d6c-4311-8f9b-3e02787a4f1a" path="/var/lib/kubelet/pods/4194cf36-5d6c-4311-8f9b-3e02787a4f1a/volumes"
Sep 09 23:55:02.645866 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:02.645779    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e6c34a8-c8da-4057-9e5d-c8877401f083","Type":"ContainerStarted","Data":"649ce9086f2ca5de3ffbb1c21f033cc5f288ee154df6a470ed420d86932380da"}
Sep 09 23:55:02.646344 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:02.645887    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e6c34a8-c8da-4057-9e5d-c8877401f083","Type":"ContainerStarted","Data":"e96c82ea6433463c3377c6e300f98425ab3a62a28f9eb8f4f33e0484874ebb23"}
Sep 09 23:55:02.646344 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:02.645930    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e6c34a8-c8da-4057-9e5d-c8877401f083","Type":"ContainerStarted","Data":"5662f604a9f54743cfa90d46e5784316973e2e8ff015ad9816557e4f4fc10487"}
Sep 09 23:55:02.646344 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:02.645974    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e6c34a8-c8da-4057-9e5d-c8877401f083","Type":"ContainerStarted","Data":"401a19b97b071bc4ef3c74b508ed107cf97c8b2cacf80405e04fa75eb34d9628"}
Sep 09 23:55:02.646344 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:02.646017    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e6c34a8-c8da-4057-9e5d-c8877401f083","Type":"ContainerStarted","Data":"4b88bfe86830f3aa1065534ba54c3bfa102d183bc82eaa5624bbe5160f86bb9a"}
Sep 09 23:55:02.646344 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:02.646078    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3e6c34a8-c8da-4057-9e5d-c8877401f083","Type":"ContainerStarted","Data":"28541ce9d483024002df2eddce5a02aa95a7aab4048b197369c9ad8fca73ed04"}
Sep 09 23:55:02.713287 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:02.713231    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.691308148 podStartE2EDuration="2.713211986s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.903838ms" isStatefulPod=false podCreationTimestamp="2026-09-09 23:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:55:02.711651548 +0000 UTC m=+382.962787616" watchObservedRunningTime="2026-09-09 23:55:02.713211986 +0000 UTC m=+382.964348029"
Sep 09 23:55:06.008967 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:55:06.008923    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:56:01.009032 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:01.008997    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:56:01.024298 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:01.024273    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:56:01.899443 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:01.899415    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 23:56:37.495449 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.495406    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"]
Sep 09 23:56:37.498819 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.498800    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.502194 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.502160    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 23:56:37.502542 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.502522    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5zk9m\""
Sep 09 23:56:37.503635 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.503592    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 23:56:37.504093 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.504076    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 23:56:37.504093 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.504082    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 23:56:37.539386 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.539356    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"]
Sep 09 23:56:37.629917 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.629890    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b105f9d5-9554-4055-9eb8-582aafbc0298-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.630020 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.629932    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b105f9d5-9554-4055-9eb8-582aafbc0298-manifests\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.630020 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.629980    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b105f9d5-9554-4055-9eb8-582aafbc0298-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.630020 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.630008    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4p2k\" (UniqueName: \"kubernetes.io/projected/b105f9d5-9554-4055-9eb8-582aafbc0298-kube-api-access-v4p2k\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.630152 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.630080    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b105f9d5-9554-4055-9eb8-582aafbc0298-tmp\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.730875 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.730848    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b105f9d5-9554-4055-9eb8-582aafbc0298-tmp\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.730978 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.730883    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b105f9d5-9554-4055-9eb8-582aafbc0298-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.730978 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.730916    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b105f9d5-9554-4055-9eb8-582aafbc0298-manifests\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.730978 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.730940    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b105f9d5-9554-4055-9eb8-582aafbc0298-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.731163 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.731083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4p2k\" (UniqueName: \"kubernetes.io/projected/b105f9d5-9554-4055-9eb8-582aafbc0298-kube-api-access-v4p2k\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.731378 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.731355    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b105f9d5-9554-4055-9eb8-582aafbc0298-manifests\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.733186 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.733161    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b105f9d5-9554-4055-9eb8-582aafbc0298-tmp\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.733556 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.733536    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b105f9d5-9554-4055-9eb8-582aafbc0298-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.733605 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.733546    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b105f9d5-9554-4055-9eb8-582aafbc0298-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.741046 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.741013    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4p2k\" (UniqueName: \"kubernetes.io/projected/b105f9d5-9554-4055-9eb8-582aafbc0298-kube-api-access-v4p2k\") pod \"opendatahub-operator-controller-manager-54cfd54796-hhlph\" (UID: \"b105f9d5-9554-4055-9eb8-582aafbc0298\") " pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:37.809494 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:37.809437    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:38.082963 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:38.082790    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"]
Sep 09 23:56:38.086007 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:56:38.085982    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb105f9d5_9554_4055_9eb8_582aafbc0298.slice/crio-c9575b8c01c4e23a8a292414a612d4eb7afa5994e8dbac188232179e8f33fd1c WatchSource:0}: Error finding container c9575b8c01c4e23a8a292414a612d4eb7afa5994e8dbac188232179e8f33fd1c: Status 404 returned error can't find the container with id c9575b8c01c4e23a8a292414a612d4eb7afa5994e8dbac188232179e8f33fd1c
Sep 09 23:56:39.032847 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:39.032798    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph" event={"ID":"b105f9d5-9554-4055-9eb8-582aafbc0298","Type":"ContainerStarted","Data":"c9575b8c01c4e23a8a292414a612d4eb7afa5994e8dbac188232179e8f33fd1c"}
Sep 09 23:56:42.049886 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:42.049851    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph" event={"ID":"b105f9d5-9554-4055-9eb8-582aafbc0298","Type":"ContainerStarted","Data":"2330064634dac5f37aed35bae3d3f0b49ebdde557cfb44b18247f58ca7309845"}
Sep 09 23:56:43.056067 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:43.056011    2556 generic.go:356] "Generic (PLEG): container finished" podID="b105f9d5-9554-4055-9eb8-582aafbc0298" containerID="2330064634dac5f37aed35bae3d3f0b49ebdde557cfb44b18247f58ca7309845" exitCode=0
Sep 09 23:56:43.056531 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:43.056100    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph" event={"ID":"b105f9d5-9554-4055-9eb8-582aafbc0298","Type":"ContainerDied","Data":"2330064634dac5f37aed35bae3d3f0b49ebdde557cfb44b18247f58ca7309845"}
Sep 09 23:56:44.062715 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:44.062674    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph" event={"ID":"b105f9d5-9554-4055-9eb8-582aafbc0298","Type":"ContainerStarted","Data":"8ca7acecb9930dba57a4ef04ebf3ec38163c3e883b1cfa5e1e2825134835cc9d"}
Sep 09 23:56:44.063136 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:44.062819    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:55.068669 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:55.068639    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph"
Sep 09 23:56:55.090130 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:55.090080    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54cfd54796-hhlph" podStartSLOduration=13.688805812 podStartE2EDuration="18.090046748s" totalImagesPullingTime="3.326913931s" totalInitContainerRuntime="1.074327005s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:56:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:56:44.108800743 +0000 UTC m=+484.359936786" watchObservedRunningTime="2026-09-09 23:56:55.090046748 +0000 UTC m=+495.341182791"
Sep 09 23:56:57.897489 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:57.897457    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"]
Sep 09 23:56:57.911784 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:57.911746    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"]
Sep 09 23:56:57.911944 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:57.911847    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:57.914553 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:57.914412    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 23:56:57.914553 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:57.914432    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 23:56:57.914875 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:57.914853    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-79cd4\""
Sep 09 23:56:57.915341 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:57.915322    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 23:56:57.915452 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:57.915375    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 23:56:57.986412 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:57.986391    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/485f4554-94b3-4e40-ab8c-aed5f078bb14-tmp\") pod \"kube-auth-proxy-7fbcb8d956-mkkn8\" (UID: \"485f4554-94b3-4e40-ab8c-aed5f078bb14\") " pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:57.986514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:57.986419    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485f4554-94b3-4e40-ab8c-aed5f078bb14-tls-certs\") pod \"kube-auth-proxy-7fbcb8d956-mkkn8\" (UID: \"485f4554-94b3-4e40-ab8c-aed5f078bb14\") " pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:57.986514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:57.986448    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp75x\" (UniqueName: \"kubernetes.io/projected/485f4554-94b3-4e40-ab8c-aed5f078bb14-kube-api-access-bp75x\") pod \"kube-auth-proxy-7fbcb8d956-mkkn8\" (UID: \"485f4554-94b3-4e40-ab8c-aed5f078bb14\") " pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:58.087321 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:58.087296    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/485f4554-94b3-4e40-ab8c-aed5f078bb14-tmp\") pod \"kube-auth-proxy-7fbcb8d956-mkkn8\" (UID: \"485f4554-94b3-4e40-ab8c-aed5f078bb14\") " pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:58.087406 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:58.087327    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485f4554-94b3-4e40-ab8c-aed5f078bb14-tls-certs\") pod \"kube-auth-proxy-7fbcb8d956-mkkn8\" (UID: \"485f4554-94b3-4e40-ab8c-aed5f078bb14\") " pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:58.087406 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:58.087358    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp75x\" (UniqueName: \"kubernetes.io/projected/485f4554-94b3-4e40-ab8c-aed5f078bb14-kube-api-access-bp75x\") pod \"kube-auth-proxy-7fbcb8d956-mkkn8\" (UID: \"485f4554-94b3-4e40-ab8c-aed5f078bb14\") " pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:58.087492 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:56:58.087436    2556 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 09 23:56:58.087533 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:56:58.087492    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/485f4554-94b3-4e40-ab8c-aed5f078bb14-tls-certs podName:485f4554-94b3-4e40-ab8c-aed5f078bb14 nodeName:}" failed. No retries permitted until 2026-09-09 23:56:58.58747515 +0000 UTC m=+498.838611170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/485f4554-94b3-4e40-ab8c-aed5f078bb14-tls-certs") pod "kube-auth-proxy-7fbcb8d956-mkkn8" (UID: "485f4554-94b3-4e40-ab8c-aed5f078bb14") : secret "kube-auth-proxy-tls" not found
Sep 09 23:56:58.089433 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:58.089407    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/485f4554-94b3-4e40-ab8c-aed5f078bb14-tmp\") pod \"kube-auth-proxy-7fbcb8d956-mkkn8\" (UID: \"485f4554-94b3-4e40-ab8c-aed5f078bb14\") " pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:58.096381 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:58.096362    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp75x\" (UniqueName: \"kubernetes.io/projected/485f4554-94b3-4e40-ab8c-aed5f078bb14-kube-api-access-bp75x\") pod \"kube-auth-proxy-7fbcb8d956-mkkn8\" (UID: \"485f4554-94b3-4e40-ab8c-aed5f078bb14\") " pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:58.590269 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:58.590225    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485f4554-94b3-4e40-ab8c-aed5f078bb14-tls-certs\") pod \"kube-auth-proxy-7fbcb8d956-mkkn8\" (UID: \"485f4554-94b3-4e40-ab8c-aed5f078bb14\") " pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:58.592438 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:58.592410    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/485f4554-94b3-4e40-ab8c-aed5f078bb14-tls-certs\") pod \"kube-auth-proxy-7fbcb8d956-mkkn8\" (UID: \"485f4554-94b3-4e40-ab8c-aed5f078bb14\") " pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:58.823974 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:58.823939    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"
Sep 09 23:56:59.096293 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:59.096214    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8"]
Sep 09 23:56:59.098575 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:56:59.098547    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485f4554_94b3_4e40_ab8c_aed5f078bb14.slice/crio-f17cb3dd1237602230753594c57b473aef606cfdbac3061ebc339c5d1adc87fa WatchSource:0}: Error finding container f17cb3dd1237602230753594c57b473aef606cfdbac3061ebc339c5d1adc87fa: Status 404 returned error can't find the container with id f17cb3dd1237602230753594c57b473aef606cfdbac3061ebc339c5d1adc87fa
Sep 09 23:56:59.128383 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:56:59.128326    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8" event={"ID":"485f4554-94b3-4e40-ab8c-aed5f078bb14","Type":"ContainerStarted","Data":"f17cb3dd1237602230753594c57b473aef606cfdbac3061ebc339c5d1adc87fa"}
Sep 09 23:57:02.143549 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:02.143517    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8" event={"ID":"485f4554-94b3-4e40-ab8c-aed5f078bb14","Type":"ContainerStarted","Data":"c42e649fc3cc4a1e23ada2e5620ad44539962a908a2f3630a38437dac28fb2f0"}
Sep 09 23:57:02.159334 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:02.159292    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7fbcb8d956-mkkn8" podStartSLOduration=2.471372529 podStartE2EDuration="5.159277132s" totalImagesPullingTime="2.687904603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:57:02.158421276 +0000 UTC m=+502.409557318" watchObservedRunningTime="2026-09-09 23:57:02.159277132 +0000 UTC m=+502.410413174"
Sep 09 23:57:08.673104 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:08.673048    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cz4kd"]
Sep 09 23:57:08.676351 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:08.676332    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd"
Sep 09 23:57:08.679010 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:08.678982    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 09 23:57:08.679010 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:08.678994    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-25qcr\""
Sep 09 23:57:08.686454 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:08.686432    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cz4kd"]
Sep 09 23:57:08.754887 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:08.754861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54466006-34a1-40ad-be74-b22d9ab53140-cert\") pod \"odh-model-controller-7998c67dc4-cz4kd\" (UID: \"54466006-34a1-40ad-be74-b22d9ab53140\") " pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd"
Sep 09 23:57:08.755006 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:08.754915    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdcgj\" (UniqueName: \"kubernetes.io/projected/54466006-34a1-40ad-be74-b22d9ab53140-kube-api-access-cdcgj\") pod \"odh-model-controller-7998c67dc4-cz4kd\" (UID: \"54466006-34a1-40ad-be74-b22d9ab53140\") " pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd"
Sep 09 23:57:08.855344 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:08.855314    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54466006-34a1-40ad-be74-b22d9ab53140-cert\") pod \"odh-model-controller-7998c67dc4-cz4kd\" (UID: \"54466006-34a1-40ad-be74-b22d9ab53140\") " pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd"
Sep 09 23:57:08.855472 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:08.855373    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdcgj\" (UniqueName: \"kubernetes.io/projected/54466006-34a1-40ad-be74-b22d9ab53140-kube-api-access-cdcgj\") pod \"odh-model-controller-7998c67dc4-cz4kd\" (UID: \"54466006-34a1-40ad-be74-b22d9ab53140\") " pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd"
Sep 09 23:57:08.855472 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:57:08.855454    2556 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 23:57:08.855576 ip-10-0-139-211 kubenswrapper[2556]: E0909 23:57:08.855515    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54466006-34a1-40ad-be74-b22d9ab53140-cert podName:54466006-34a1-40ad-be74-b22d9ab53140 nodeName:}" failed. No retries permitted until 2026-09-09 23:57:09.355497921 +0000 UTC m=+509.606633944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/54466006-34a1-40ad-be74-b22d9ab53140-cert") pod "odh-model-controller-7998c67dc4-cz4kd" (UID: "54466006-34a1-40ad-be74-b22d9ab53140") : secret "odh-model-controller-webhook-cert" not found
Sep 09 23:57:08.863625 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:08.863600    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdcgj\" (UniqueName: \"kubernetes.io/projected/54466006-34a1-40ad-be74-b22d9ab53140-kube-api-access-cdcgj\") pod \"odh-model-controller-7998c67dc4-cz4kd\" (UID: \"54466006-34a1-40ad-be74-b22d9ab53140\") " pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd"
Sep 09 23:57:09.358790 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:09.358749    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54466006-34a1-40ad-be74-b22d9ab53140-cert\") pod \"odh-model-controller-7998c67dc4-cz4kd\" (UID: \"54466006-34a1-40ad-be74-b22d9ab53140\") " pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd"
Sep 09 23:57:09.361208 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:09.361184    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54466006-34a1-40ad-be74-b22d9ab53140-cert\") pod \"odh-model-controller-7998c67dc4-cz4kd\" (UID: \"54466006-34a1-40ad-be74-b22d9ab53140\") " pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd"
Sep 09 23:57:09.587090 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:09.587045    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd"
Sep 09 23:57:09.855442 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:09.855149    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-cz4kd"]
Sep 09 23:57:09.858994 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:57:09.858966    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54466006_34a1_40ad_be74_b22d9ab53140.slice/crio-0a5589371be5d6effc379d6502870054fe1173bdd9b9d9da3b76353a2a9bba2c WatchSource:0}: Error finding container 0a5589371be5d6effc379d6502870054fe1173bdd9b9d9da3b76353a2a9bba2c: Status 404 returned error can't find the container with id 0a5589371be5d6effc379d6502870054fe1173bdd9b9d9da3b76353a2a9bba2c
Sep 09 23:57:10.185840 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:10.185807    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd" event={"ID":"54466006-34a1-40ad-be74-b22d9ab53140","Type":"ContainerStarted","Data":"0a5589371be5d6effc379d6502870054fe1173bdd9b9d9da3b76353a2a9bba2c"}
Sep 09 23:57:13.206021 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.205984    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd" event={"ID":"54466006-34a1-40ad-be74-b22d9ab53140","Type":"ContainerStarted","Data":"a21800e939d6349cf8a6a5260074ac0b4bfb96b5267fd2d6d3e5ebeaad26179a"}
Sep 09 23:57:13.206477 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.206120    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd"
Sep 09 23:57:13.222894 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.222847    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd" podStartSLOduration=2.546436307 podStartE2EDuration="5.222834321s" totalImagesPullingTime="2.676398014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:57:13.222311365 +0000 UTC m=+513.473447407" watchObservedRunningTime="2026-09-09 23:57:13.222834321 +0000 UTC m=+513.473970364"
Sep 09 23:57:13.982648 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.982608    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"]
Sep 09 23:57:13.985686 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.985665    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:13.988157 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.988134    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 23:57:13.988267 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.988134    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 23:57:13.989002 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.988977    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 23:57:13.989002 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.988999    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 23:57:13.989179 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.989011    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 23:57:13.989179 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.989044    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ggdqv\""
Sep 09 23:57:13.991461 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.991430    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-manager-config\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:13.991541 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.991475    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-cert\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:13.991601 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.991585    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-metrics-cert\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:13.991646 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.991611    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkfl\" (UniqueName: \"kubernetes.io/projected/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-kube-api-access-9pkfl\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:13.995085 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:13.995049    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"]
Sep 09 23:57:14.092227 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:14.092201    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-cert\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:14.092328 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:14.092258    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-metrics-cert\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:14.092328 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:14.092279    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkfl\" (UniqueName: \"kubernetes.io/projected/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-kube-api-access-9pkfl\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:14.092328 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:14.092310    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-manager-config\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:14.093047 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:14.093016    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-manager-config\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:14.094596 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:14.094579    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-cert\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:14.094667 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:14.094629    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-metrics-cert\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:14.099551 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:14.099531    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkfl\" (UniqueName: \"kubernetes.io/projected/a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e-kube-api-access-9pkfl\") pod \"lws-controller-manager-5895cdb77f-tmnbn\" (UID: \"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e\") " pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:14.295651 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:14.295586    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:14.563914 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:14.563889    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"]
Sep 09 23:57:14.565931 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:57:14.565906    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fc01d1_f577_42bf_a1f4_decd0a3e2c6e.slice/crio-bdcd5cbcfbc2764daac5274ed9255ead2c05c690e9b72eccb3ab22cc5e1eaea9 WatchSource:0}: Error finding container bdcd5cbcfbc2764daac5274ed9255ead2c05c690e9b72eccb3ab22cc5e1eaea9: Status 404 returned error can't find the container with id bdcd5cbcfbc2764daac5274ed9255ead2c05c690e9b72eccb3ab22cc5e1eaea9
Sep 09 23:57:15.217172 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:15.217102    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn" event={"ID":"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e","Type":"ContainerStarted","Data":"bdcd5cbcfbc2764daac5274ed9255ead2c05c690e9b72eccb3ab22cc5e1eaea9"}
Sep 09 23:57:17.228937 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:17.228902    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn" event={"ID":"a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e","Type":"ContainerStarted","Data":"016c25013381fd55611cb93441b33e3a9c4512e69779fafea30d87579a3e3a24"}
Sep 09 23:57:17.229422 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:17.229018    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:57:17.249091 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:17.249023    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn" podStartSLOduration=2.129732516 podStartE2EDuration="4.249008902s" totalImagesPullingTime="2.119276386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:57:17.248093507 +0000 UTC m=+517.499229552" watchObservedRunningTime="2026-09-09 23:57:17.249008902 +0000 UTC m=+517.500144938"
Sep 09 23:57:24.212037 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:24.212004    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-cz4kd"
Sep 09 23:57:28.235307 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:57:28.235277    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5895cdb77f-tmnbn"
Sep 09 23:58:57.778006 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:57.777971    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fbw9v"]
Sep 09 23:58:57.781389 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:57.781365    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-fbw9v"
Sep 09 23:58:57.783514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:57.783490    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 23:58:57.783514 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:57.783501    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nl2q2\""
Sep 09 23:58:57.783747 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:57.783730    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 23:58:57.793652 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:57.793626    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fbw9v"]
Sep 09 23:58:57.888472 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:57.888443    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6j4v\" (UniqueName: \"kubernetes.io/projected/97caa56d-ac6b-4229-8ca6-bcde14491e2e-kube-api-access-v6j4v\") pod \"authorino-operator-76b478bd78-fbw9v\" (UID: \"97caa56d-ac6b-4229-8ca6-bcde14491e2e\") " pod="kuadrant-system/authorino-operator-76b478bd78-fbw9v"
Sep 09 23:58:57.989050 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:57.989017    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6j4v\" (UniqueName: \"kubernetes.io/projected/97caa56d-ac6b-4229-8ca6-bcde14491e2e-kube-api-access-v6j4v\") pod \"authorino-operator-76b478bd78-fbw9v\" (UID: \"97caa56d-ac6b-4229-8ca6-bcde14491e2e\") " pod="kuadrant-system/authorino-operator-76b478bd78-fbw9v"
Sep 09 23:58:57.998914 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:57.998888    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6j4v\" (UniqueName: \"kubernetes.io/projected/97caa56d-ac6b-4229-8ca6-bcde14491e2e-kube-api-access-v6j4v\") pod \"authorino-operator-76b478bd78-fbw9v\" (UID: \"97caa56d-ac6b-4229-8ca6-bcde14491e2e\") " pod="kuadrant-system/authorino-operator-76b478bd78-fbw9v"
Sep 09 23:58:58.092249 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:58.092177    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-fbw9v"
Sep 09 23:58:58.390356 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:58.390331    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fbw9v"]
Sep 09 23:58:58.392583 ip-10-0-139-211 kubenswrapper[2556]: W0909 23:58:58.392554    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97caa56d_ac6b_4229_8ca6_bcde14491e2e.slice/crio-88ae5118988fe48382fc9d2dad9290d15c53e4b7b0e8ac633c889ed33b7bcaac WatchSource:0}: Error finding container 88ae5118988fe48382fc9d2dad9290d15c53e4b7b0e8ac633c889ed33b7bcaac: Status 404 returned error can't find the container with id 88ae5118988fe48382fc9d2dad9290d15c53e4b7b0e8ac633c889ed33b7bcaac
Sep 09 23:58:58.394851 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:58.394835    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 23:58:58.692240 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:58:58.692167    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-fbw9v" event={"ID":"97caa56d-ac6b-4229-8ca6-bcde14491e2e","Type":"ContainerStarted","Data":"88ae5118988fe48382fc9d2dad9290d15c53e4b7b0e8ac633c889ed33b7bcaac"}
Sep 09 23:59:01.707653 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:59:01.707619    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-fbw9v" event={"ID":"97caa56d-ac6b-4229-8ca6-bcde14491e2e","Type":"ContainerStarted","Data":"d90375ff875c0e852d33964cf58613b160e25731e6a6d24fa500331e4ed20de4"}
Sep 09 23:59:01.708141 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:59:01.707753    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-fbw9v"
Sep 09 23:59:01.730896 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:59:01.730838    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-fbw9v" podStartSLOduration=2.371848436 podStartE2EDuration="4.730822773s" totalImagesPullingTime="2.358974337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:59:01.728983806 +0000 UTC m=+621.980119849" watchObservedRunningTime="2026-09-09 23:59:01.730822773 +0000 UTC m=+621.981958815"
Sep 09 23:59:12.714977 ip-10-0-139-211 kubenswrapper[2556]: I0909 23:59:12.714900    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-fbw9v"
Sep 10 00:00:06.558900 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.558867    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqnm5"]
Sep 10 00:00:06.562112 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.562094    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5"
Sep 10 00:00:06.564352 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.564318    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fwrps\""
Sep 10 00:00:06.568373 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.568351    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqnm5"]
Sep 10 00:00:06.608014 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.607990    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfgf2\" (UniqueName: \"kubernetes.io/projected/06c1dbc6-2568-4490-aef9-c739445141de-kube-api-access-jfgf2\") pod \"authorino-7bcfd8f9cd-xqnm5\" (UID: \"06c1dbc6-2568-4490-aef9-c739445141de\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5"
Sep 10 00:00:06.708800 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.708776    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfgf2\" (UniqueName: \"kubernetes.io/projected/06c1dbc6-2568-4490-aef9-c739445141de-kube-api-access-jfgf2\") pod \"authorino-7bcfd8f9cd-xqnm5\" (UID: \"06c1dbc6-2568-4490-aef9-c739445141de\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5"
Sep 10 00:00:06.719603 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.719584    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfgf2\" (UniqueName: \"kubernetes.io/projected/06c1dbc6-2568-4490-aef9-c739445141de-kube-api-access-jfgf2\") pod \"authorino-7bcfd8f9cd-xqnm5\" (UID: \"06c1dbc6-2568-4490-aef9-c739445141de\") " pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5"
Sep 10 00:00:06.731559 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.731535    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-pzlmj"]
Sep 10 00:00:06.734702 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.734685    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pzlmj"
Sep 10 00:00:06.738358 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.738339    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pzlmj"]
Sep 10 00:00:06.809871 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.809814    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw68z\" (UniqueName: \"kubernetes.io/projected/e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82-kube-api-access-mw68z\") pod \"authorino-59c974bd48-pzlmj\" (UID: \"e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82\") " pod="kuadrant-system/authorino-59c974bd48-pzlmj"
Sep 10 00:00:06.905557 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.905539    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5"
Sep 10 00:00:06.910516 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.910490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw68z\" (UniqueName: \"kubernetes.io/projected/e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82-kube-api-access-mw68z\") pod \"authorino-59c974bd48-pzlmj\" (UID: \"e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82\") " pod="kuadrant-system/authorino-59c974bd48-pzlmj"
Sep 10 00:00:06.918482 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:06.918454    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw68z\" (UniqueName: \"kubernetes.io/projected/e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82-kube-api-access-mw68z\") pod \"authorino-59c974bd48-pzlmj\" (UID: \"e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82\") " pod="kuadrant-system/authorino-59c974bd48-pzlmj"
Sep 10 00:00:07.045659 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:07.045630    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pzlmj"
Sep 10 00:00:07.170009 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:07.169978    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqnm5"]
Sep 10 00:00:07.172477 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:00:07.172448    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06c1dbc6_2568_4490_aef9_c739445141de.slice/crio-7c6727d623a56ec1565c2fc0a53f8e2687bb36d588a3ed67e8533374398f0d76 WatchSource:0}: Error finding container 7c6727d623a56ec1565c2fc0a53f8e2687bb36d588a3ed67e8533374398f0d76: Status 404 returned error can't find the container with id 7c6727d623a56ec1565c2fc0a53f8e2687bb36d588a3ed67e8533374398f0d76
Sep 10 00:00:07.309898 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:07.309874    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pzlmj"]
Sep 10 00:00:07.311716 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:00:07.311691    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3841fc0_b0ef_4a65_9c78_a29fb3f1ca82.slice/crio-67c1562903a95134f59ac7929bcb3204ee4436e1cd11c88479d4a8cbd427c56e WatchSource:0}: Error finding container 67c1562903a95134f59ac7929bcb3204ee4436e1cd11c88479d4a8cbd427c56e: Status 404 returned error can't find the container with id 67c1562903a95134f59ac7929bcb3204ee4436e1cd11c88479d4a8cbd427c56e
Sep 10 00:00:08.023051 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:08.023014    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pzlmj" event={"ID":"e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82","Type":"ContainerStarted","Data":"67c1562903a95134f59ac7929bcb3204ee4436e1cd11c88479d4a8cbd427c56e"}
Sep 10 00:00:08.024249 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:08.024223    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5" event={"ID":"06c1dbc6-2568-4490-aef9-c739445141de","Type":"ContainerStarted","Data":"7c6727d623a56ec1565c2fc0a53f8e2687bb36d588a3ed67e8533374398f0d76"}
Sep 10 00:00:16.062855 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:16.062821    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pzlmj" event={"ID":"e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82","Type":"ContainerStarted","Data":"9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a"}
Sep 10 00:00:16.064343 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:16.064316    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5" event={"ID":"06c1dbc6-2568-4490-aef9-c739445141de","Type":"ContainerStarted","Data":"455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61"}
Sep 10 00:00:16.076873 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:16.076834    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-pzlmj" podStartSLOduration=1.8211037079999999 podStartE2EDuration="10.076818477s" totalImagesPullingTime="8.255714769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:00:16.076559194 +0000 UTC m=+696.327695239" watchObservedRunningTime="2026-09-10 00:00:16.076818477 +0000 UTC m=+696.327954521"
Sep 10 00:00:16.109585 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:16.109534    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5" podStartSLOduration=1.71017129 podStartE2EDuration="10.109517668s" totalImagesPullingTime="8.399346378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:00:16.089654101 +0000 UTC m=+696.340790146" watchObservedRunningTime="2026-09-10 00:00:16.109517668 +0000 UTC m=+696.360653710"
Sep 10 00:00:16.110980 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:16.110957    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqnm5"]
Sep 10 00:00:18.073593 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:18.073552    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5" podUID="06c1dbc6-2568-4490-aef9-c739445141de" containerName="authorino" containerID="cri-o://455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61" gracePeriod=30
Sep 10 00:00:18.312942 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:18.312919    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5"
Sep 10 00:00:18.399600 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:18.399536    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfgf2\" (UniqueName: \"kubernetes.io/projected/06c1dbc6-2568-4490-aef9-c739445141de-kube-api-access-jfgf2\") pod \"06c1dbc6-2568-4490-aef9-c739445141de\" (UID: \"06c1dbc6-2568-4490-aef9-c739445141de\") "
Sep 10 00:00:18.401633 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:18.401604    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c1dbc6-2568-4490-aef9-c739445141de-kube-api-access-jfgf2" (OuterVolumeSpecName: "kube-api-access-jfgf2") pod "06c1dbc6-2568-4490-aef9-c739445141de" (UID: "06c1dbc6-2568-4490-aef9-c739445141de"). InnerVolumeSpecName "kube-api-access-jfgf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:00:18.500367 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:18.500337    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfgf2\" (UniqueName: \"kubernetes.io/projected/06c1dbc6-2568-4490-aef9-c739445141de-kube-api-access-jfgf2\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:00:19.078762 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:19.078727    2556 generic.go:356] "Generic (PLEG): container finished" podID="06c1dbc6-2568-4490-aef9-c739445141de" containerID="455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61" exitCode=0
Sep 10 00:00:19.078762 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:19.078765    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5"
Sep 10 00:00:19.079239 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:19.078793    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5" event={"ID":"06c1dbc6-2568-4490-aef9-c739445141de","Type":"ContainerDied","Data":"455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61"}
Sep 10 00:00:19.079239 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:19.078831    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-xqnm5" event={"ID":"06c1dbc6-2568-4490-aef9-c739445141de","Type":"ContainerDied","Data":"7c6727d623a56ec1565c2fc0a53f8e2687bb36d588a3ed67e8533374398f0d76"}
Sep 10 00:00:19.079239 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:19.078855    2556 scope.go:117] "RemoveContainer" containerID="455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61"
Sep 10 00:00:19.089753 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:19.089738    2556 scope.go:117] "RemoveContainer" containerID="455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61"
Sep 10 00:00:19.090029 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:00:19.090010    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61\": container with ID starting with 455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61 not found: ID does not exist" containerID="455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61"
Sep 10 00:00:19.090097 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:19.090036    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61"} err="failed to get container status \"455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61\": rpc error: code = NotFound desc = could not find container \"455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61\": container with ID starting with 455d4fe7baf5be16247f77a3d8b9399eb24a936c796493e2b0919f875d13df61 not found: ID does not exist"
Sep 10 00:00:19.099785 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:19.099765    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqnm5"]
Sep 10 00:00:19.103570 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:19.103538    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-xqnm5"]
Sep 10 00:00:20.267372 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:20.267342    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06c1dbc6-2568-4490-aef9-c739445141de" path="/var/lib/kubelet/pods/06c1dbc6-2568-4490-aef9-c739445141de/volumes"
Sep 10 00:00:39.738381 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.738339    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-s4k5g"]
Sep 10 00:00:39.738845 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.738698    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c1dbc6-2568-4490-aef9-c739445141de" containerName="authorino"
Sep 10 00:00:39.738845 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.738711    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c1dbc6-2568-4490-aef9-c739445141de" containerName="authorino"
Sep 10 00:00:39.738845 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.738785    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c1dbc6-2568-4490-aef9-c739445141de" containerName="authorino"
Sep 10 00:00:39.741549 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.741528    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-s4k5g"
Sep 10 00:00:39.743982 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.743966    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 00:00:39.748776 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.748753    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-s4k5g"]
Sep 10 00:00:39.865303 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.865266    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-s4k5g"]
Sep 10 00:00:39.865563 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:00:39.865538    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hbvdw tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-s4k5g" podUID="c7043730-18d3-4984-9c04-06f1b6076721"
Sep 10 00:00:39.878946 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.878914    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbvdw\" (UniqueName: \"kubernetes.io/projected/c7043730-18d3-4984-9c04-06f1b6076721-kube-api-access-hbvdw\") pod \"authorino-6d4dfc649f-s4k5g\" (UID: \"c7043730-18d3-4984-9c04-06f1b6076721\") " pod="kuadrant-system/authorino-6d4dfc649f-s4k5g"
Sep 10 00:00:39.879123 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.878966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7043730-18d3-4984-9c04-06f1b6076721-tls-cert\") pod \"authorino-6d4dfc649f-s4k5g\" (UID: \"c7043730-18d3-4984-9c04-06f1b6076721\") " pod="kuadrant-system/authorino-6d4dfc649f-s4k5g"
Sep 10 00:00:39.888292 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.888271    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-fd7564778-28jfc"]
Sep 10 00:00:39.891537 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.891520    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd7564778-28jfc"
Sep 10 00:00:39.897724 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.897700    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fd7564778-28jfc"]
Sep 10 00:00:39.980136 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.980108    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbvdw\" (UniqueName: \"kubernetes.io/projected/c7043730-18d3-4984-9c04-06f1b6076721-kube-api-access-hbvdw\") pod \"authorino-6d4dfc649f-s4k5g\" (UID: \"c7043730-18d3-4984-9c04-06f1b6076721\") " pod="kuadrant-system/authorino-6d4dfc649f-s4k5g"
Sep 10 00:00:39.980264 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.980158    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7043730-18d3-4984-9c04-06f1b6076721-tls-cert\") pod \"authorino-6d4dfc649f-s4k5g\" (UID: \"c7043730-18d3-4984-9c04-06f1b6076721\") " pod="kuadrant-system/authorino-6d4dfc649f-s4k5g"
Sep 10 00:00:39.982450 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.982430    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7043730-18d3-4984-9c04-06f1b6076721-tls-cert\") pod \"authorino-6d4dfc649f-s4k5g\" (UID: \"c7043730-18d3-4984-9c04-06f1b6076721\") " pod="kuadrant-system/authorino-6d4dfc649f-s4k5g"
Sep 10 00:00:39.986895 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:39.986877    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbvdw\" (UniqueName: \"kubernetes.io/projected/c7043730-18d3-4984-9c04-06f1b6076721-kube-api-access-hbvdw\") pod \"authorino-6d4dfc649f-s4k5g\" (UID: \"c7043730-18d3-4984-9c04-06f1b6076721\") " pod="kuadrant-system/authorino-6d4dfc649f-s4k5g"
Sep 10 00:00:40.081203 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.081123    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n6h5\" (UniqueName: \"kubernetes.io/projected/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-kube-api-access-7n6h5\") pod \"authorino-fd7564778-28jfc\" (UID: \"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0\") " pod="kuadrant-system/authorino-fd7564778-28jfc"
Sep 10 00:00:40.081203 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.081200    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-tls-cert\") pod \"authorino-fd7564778-28jfc\" (UID: \"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0\") " pod="kuadrant-system/authorino-fd7564778-28jfc"
Sep 10 00:00:40.172695 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.172668    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-s4k5g"
Sep 10 00:00:40.181336 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.181316    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-s4k5g"
Sep 10 00:00:40.181666 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.181652    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n6h5\" (UniqueName: \"kubernetes.io/projected/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-kube-api-access-7n6h5\") pod \"authorino-fd7564778-28jfc\" (UID: \"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0\") " pod="kuadrant-system/authorino-fd7564778-28jfc"
Sep 10 00:00:40.181709 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.181699    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-tls-cert\") pod \"authorino-fd7564778-28jfc\" (UID: \"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0\") " pod="kuadrant-system/authorino-fd7564778-28jfc"
Sep 10 00:00:40.184064 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.184032    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-tls-cert\") pod \"authorino-fd7564778-28jfc\" (UID: \"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0\") " pod="kuadrant-system/authorino-fd7564778-28jfc"
Sep 10 00:00:40.190025 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.190004    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n6h5\" (UniqueName: \"kubernetes.io/projected/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-kube-api-access-7n6h5\") pod \"authorino-fd7564778-28jfc\" (UID: \"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0\") " pod="kuadrant-system/authorino-fd7564778-28jfc"
Sep 10 00:00:40.201110 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.201089    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd7564778-28jfc"
Sep 10 00:00:40.283002 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.282971    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbvdw\" (UniqueName: \"kubernetes.io/projected/c7043730-18d3-4984-9c04-06f1b6076721-kube-api-access-hbvdw\") pod \"c7043730-18d3-4984-9c04-06f1b6076721\" (UID: \"c7043730-18d3-4984-9c04-06f1b6076721\") "
Sep 10 00:00:40.283152 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.283017    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7043730-18d3-4984-9c04-06f1b6076721-tls-cert\") pod \"c7043730-18d3-4984-9c04-06f1b6076721\" (UID: \"c7043730-18d3-4984-9c04-06f1b6076721\") "
Sep 10 00:00:40.284939 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.284914    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7043730-18d3-4984-9c04-06f1b6076721-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c7043730-18d3-4984-9c04-06f1b6076721" (UID: "c7043730-18d3-4984-9c04-06f1b6076721"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:00:40.285076 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.284996    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7043730-18d3-4984-9c04-06f1b6076721-kube-api-access-hbvdw" (OuterVolumeSpecName: "kube-api-access-hbvdw") pod "c7043730-18d3-4984-9c04-06f1b6076721" (UID: "c7043730-18d3-4984-9c04-06f1b6076721"). InnerVolumeSpecName "kube-api-access-hbvdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:00:40.383640 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.383559    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbvdw\" (UniqueName: \"kubernetes.io/projected/c7043730-18d3-4984-9c04-06f1b6076721-kube-api-access-hbvdw\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:00:40.383640 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.383586    2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7043730-18d3-4984-9c04-06f1b6076721-tls-cert\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:00:40.468352 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:40.468327    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-fd7564778-28jfc"]
Sep 10 00:00:40.470566 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:00:40.470542    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1c0e6d5_fe74_4a44_b9b4_e906f17e04e0.slice/crio-29f5f88a67793eed0b5e60d671b5ce55a43cf1318acd7a59b571ea747c36a9a1 WatchSource:0}: Error finding container 29f5f88a67793eed0b5e60d671b5ce55a43cf1318acd7a59b571ea747c36a9a1: Status 404 returned error can't find the container with id 29f5f88a67793eed0b5e60d671b5ce55a43cf1318acd7a59b571ea747c36a9a1
Sep 10 00:00:41.179999 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.179969    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-s4k5g"
Sep 10 00:00:41.180354 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.179963    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd7564778-28jfc" event={"ID":"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0","Type":"ContainerStarted","Data":"5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c"}
Sep 10 00:00:41.180354 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.180046    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd7564778-28jfc" event={"ID":"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0","Type":"ContainerStarted","Data":"29f5f88a67793eed0b5e60d671b5ce55a43cf1318acd7a59b571ea747c36a9a1"}
Sep 10 00:00:41.197996 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.197951    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-fd7564778-28jfc" podStartSLOduration=1.561670049 podStartE2EDuration="2.197936717s" totalImagesPullingTime="636.266668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:00:41.196907162 +0000 UTC m=+721.448043206" watchObservedRunningTime="2026-09-10 00:00:41.197936717 +0000 UTC m=+721.449072760"
Sep 10 00:00:41.219824 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.219791    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-s4k5g"]
Sep 10 00:00:41.225417 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.225383    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pzlmj"]
Sep 10 00:00:41.225610 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.225584    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-pzlmj" podUID="e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82" containerName="authorino" containerID="cri-o://9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a" gracePeriod=30
Sep 10 00:00:41.227614 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.227591    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-s4k5g"]
Sep 10 00:00:41.457945 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.457921    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pzlmj"
Sep 10 00:00:41.491794 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.491762    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw68z\" (UniqueName: \"kubernetes.io/projected/e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82-kube-api-access-mw68z\") pod \"e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82\" (UID: \"e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82\") "
Sep 10 00:00:41.493777 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.493747    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82-kube-api-access-mw68z" (OuterVolumeSpecName: "kube-api-access-mw68z") pod "e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82" (UID: "e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82"). InnerVolumeSpecName "kube-api-access-mw68z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:00:41.592798 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:41.592709    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw68z\" (UniqueName: \"kubernetes.io/projected/e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82-kube-api-access-mw68z\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:00:42.185562 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:42.185527    2556 generic.go:356] "Generic (PLEG): container finished" podID="e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82" containerID="9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a" exitCode=0
Sep 10 00:00:42.185966 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:42.185567    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pzlmj"
Sep 10 00:00:42.185966 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:42.185610    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pzlmj" event={"ID":"e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82","Type":"ContainerDied","Data":"9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a"}
Sep 10 00:00:42.185966 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:42.185651    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pzlmj" event={"ID":"e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82","Type":"ContainerDied","Data":"67c1562903a95134f59ac7929bcb3204ee4436e1cd11c88479d4a8cbd427c56e"}
Sep 10 00:00:42.185966 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:42.185673    2556 scope.go:117] "RemoveContainer" containerID="9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a"
Sep 10 00:00:42.194852 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:42.194836    2556 scope.go:117] "RemoveContainer" containerID="9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a"
Sep 10 00:00:42.195137 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:00:42.195116    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a\": container with ID starting with 9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a not found: ID does not exist" containerID="9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a"
Sep 10 00:00:42.195188 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:42.195152    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a"} err="failed to get container status \"9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a\": rpc error: code = NotFound desc = could not find container \"9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a\": container with ID starting with 9414c6598f9b5d083e5f88d922b5dde5d5403298bad25a058606efdc03e18b5a not found: ID does not exist"
Sep 10 00:00:42.205600 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:42.205577    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pzlmj"]
Sep 10 00:00:42.208312 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:42.208294    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pzlmj"]
Sep 10 00:00:42.269801 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:42.269735    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7043730-18d3-4984-9c04-06f1b6076721" path="/var/lib/kubelet/pods/c7043730-18d3-4984-9c04-06f1b6076721/volumes"
Sep 10 00:00:42.269993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:42.269980    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82" path="/var/lib/kubelet/pods/e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82/volumes"
Sep 10 00:00:45.944459 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:45.944418    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f8584cb86-wmmzm"]
Sep 10 00:00:45.944953 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:45.944932    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82" containerName="authorino"
Sep 10 00:00:45.945040 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:45.944965    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82" containerName="authorino"
Sep 10 00:00:45.945135 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:45.945093    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3841fc0-b0ef-4a65-9c78-a29fb3f1ca82" containerName="authorino"
Sep 10 00:00:45.952651 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:45.952627    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:45.955030 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:45.955001    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 00:00:45.956270 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:45.956019    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 00:00:45.956270 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:45.956037    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cx96n\""
Sep 10 00:00:45.956270 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:45.956161    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 00:00:45.958391 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:45.958372    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-wmmzm"]
Sep 10 00:00:46.032505 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.032475    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-webhook-cert\") pod \"maas-controller-6f8584cb86-wmmzm\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") " pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:46.032681 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.032533    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-metrics-certs\") pod \"maas-controller-6f8584cb86-wmmzm\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") " pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:46.032681 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.032609    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbw4t\" (UniqueName: \"kubernetes.io/projected/4d738e4c-0a56-40dd-af76-dbfac13e2dae-kube-api-access-kbw4t\") pod \"maas-controller-6f8584cb86-wmmzm\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") " pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:46.133976 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.133938    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-metrics-certs\") pod \"maas-controller-6f8584cb86-wmmzm\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") " pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:46.134186 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.133990    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbw4t\" (UniqueName: \"kubernetes.io/projected/4d738e4c-0a56-40dd-af76-dbfac13e2dae-kube-api-access-kbw4t\") pod \"maas-controller-6f8584cb86-wmmzm\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") " pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:46.134186 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.134030    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-webhook-cert\") pod \"maas-controller-6f8584cb86-wmmzm\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") " pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:46.134186 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:00:46.134129    2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 00:00:46.134350 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:00:46.134225    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-metrics-certs podName:4d738e4c-0a56-40dd-af76-dbfac13e2dae nodeName:}" failed. No retries permitted until 2026-09-10 00:00:46.634202594 +0000 UTC m=+726.885338616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-metrics-certs") pod "maas-controller-6f8584cb86-wmmzm" (UID: "4d738e4c-0a56-40dd-af76-dbfac13e2dae") : secret "maas-controller-metrics-tls" not found
Sep 10 00:00:46.134350 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:00:46.134138    2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 10 00:00:46.134350 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:00:46.134266    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-webhook-cert podName:4d738e4c-0a56-40dd-af76-dbfac13e2dae nodeName:}" failed. No retries permitted until 2026-09-10 00:00:46.634256976 +0000 UTC m=+726.885392999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-webhook-cert") pod "maas-controller-6f8584cb86-wmmzm" (UID: "4d738e4c-0a56-40dd-af76-dbfac13e2dae") : secret "maas-controller-webhook-cert" not found
Sep 10 00:00:46.142045 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.142018    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbw4t\" (UniqueName: \"kubernetes.io/projected/4d738e4c-0a56-40dd-af76-dbfac13e2dae-kube-api-access-kbw4t\") pod \"maas-controller-6f8584cb86-wmmzm\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") " pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:46.639584 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.639546    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-webhook-cert\") pod \"maas-controller-6f8584cb86-wmmzm\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") " pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:46.639776 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.639612    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-metrics-certs\") pod \"maas-controller-6f8584cb86-wmmzm\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") " pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:46.642045 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.642012    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-webhook-cert\") pod \"maas-controller-6f8584cb86-wmmzm\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") " pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:46.642180 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.642081    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-metrics-certs\") pod \"maas-controller-6f8584cb86-wmmzm\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") " pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:46.864255 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:46.864220    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:47.136679 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:47.136654    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-wmmzm"]
Sep 10 00:00:47.138804 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:00:47.138775    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d738e4c_0a56_40dd_af76_dbfac13e2dae.slice/crio-e207b89f4cdb671de5f006135d37d3093b858a6b7fc518e81bbee5567955faf3 WatchSource:0}: Error finding container e207b89f4cdb671de5f006135d37d3093b858a6b7fc518e81bbee5567955faf3: Status 404 returned error can't find the container with id e207b89f4cdb671de5f006135d37d3093b858a6b7fc518e81bbee5567955faf3
Sep 10 00:00:47.210827 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:47.210794    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f8584cb86-wmmzm" event={"ID":"4d738e4c-0a56-40dd-af76-dbfac13e2dae","Type":"ContainerStarted","Data":"e207b89f4cdb671de5f006135d37d3093b858a6b7fc518e81bbee5567955faf3"}
Sep 10 00:00:50.227344 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:50.227313    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f8584cb86-wmmzm" event={"ID":"4d738e4c-0a56-40dd-af76-dbfac13e2dae","Type":"ContainerStarted","Data":"3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9"}
Sep 10 00:00:50.227720 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:50.227372    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:00:50.246559 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:00:50.246478    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f8584cb86-wmmzm" podStartSLOduration=2.244905739 podStartE2EDuration="5.24646336s" totalImagesPullingTime="3.001557621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:00:50.2454443 +0000 UTC m=+730.496580344" watchObservedRunningTime="2026-09-10 00:00:50.24646336 +0000 UTC m=+730.497599403"
Sep 10 00:01:01.233678 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:01.233639    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:01:23.186860 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.186820    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"]
Sep 10 00:01:23.195513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.195477    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"
Sep 10 00:01:23.198466 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.198413    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 00:01:23.200016 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.199870    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"]
Sep 10 00:01:23.361771 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.361727    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2dtj\" (UniqueName: \"kubernetes.io/projected/a5f0c922-b35c-4fe8-8c67-2365c774e22f-kube-api-access-x2dtj\") pod \"payload-pre-processing-59fbc47bf8-f5g58\" (UID: \"a5f0c922-b35c-4fe8-8c67-2365c774e22f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"
Sep 10 00:01:23.361771 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.361776    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f0c922-b35c-4fe8-8c67-2365c774e22f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-f5g58\" (UID: \"a5f0c922-b35c-4fe8-8c67-2365c774e22f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"
Sep 10 00:01:23.463431 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.463393    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f0c922-b35c-4fe8-8c67-2365c774e22f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-f5g58\" (UID: \"a5f0c922-b35c-4fe8-8c67-2365c774e22f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"
Sep 10 00:01:23.463605 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.463564    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2dtj\" (UniqueName: \"kubernetes.io/projected/a5f0c922-b35c-4fe8-8c67-2365c774e22f-kube-api-access-x2dtj\") pod \"payload-pre-processing-59fbc47bf8-f5g58\" (UID: \"a5f0c922-b35c-4fe8-8c67-2365c774e22f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"
Sep 10 00:01:23.464160 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.464140    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f0c922-b35c-4fe8-8c67-2365c774e22f-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-f5g58\" (UID: \"a5f0c922-b35c-4fe8-8c67-2365c774e22f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"
Sep 10 00:01:23.471258 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.471230    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2dtj\" (UniqueName: \"kubernetes.io/projected/a5f0c922-b35c-4fe8-8c67-2365c774e22f-kube-api-access-x2dtj\") pod \"payload-pre-processing-59fbc47bf8-f5g58\" (UID: \"a5f0c922-b35c-4fe8-8c67-2365c774e22f\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"
Sep 10 00:01:23.510018 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.509986    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"
Sep 10 00:01:23.794118 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:23.794091    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"]
Sep 10 00:01:23.796735 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:01:23.796709    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f0c922_b35c_4fe8_8c67_2365c774e22f.slice/crio-ec768f1ba14a0eeae1f163887b43f92b2e90e571811b611dccdcd39f15b1518f WatchSource:0}: Error finding container ec768f1ba14a0eeae1f163887b43f92b2e90e571811b611dccdcd39f15b1518f: Status 404 returned error can't find the container with id ec768f1ba14a0eeae1f163887b43f92b2e90e571811b611dccdcd39f15b1518f
Sep 10 00:01:24.390633 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:24.390592    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58" event={"ID":"a5f0c922-b35c-4fe8-8c67-2365c774e22f","Type":"ContainerStarted","Data":"ec768f1ba14a0eeae1f163887b43f92b2e90e571811b611dccdcd39f15b1518f"}
Sep 10 00:01:27.408210 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:27.408174    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58" event={"ID":"a5f0c922-b35c-4fe8-8c67-2365c774e22f","Type":"ContainerStarted","Data":"c4590258b9685d5836daf2d84f9029b15117ae52e72aa895a4f60ee9658e67ae"}
Sep 10 00:01:27.408553 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:27.408308    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"
Sep 10 00:01:27.424168 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:27.424126    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58" podStartSLOduration=1.43126019 podStartE2EDuration="4.424112218s" totalImagesPullingTime="2.992852028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:01:27.422299648 +0000 UTC m=+767.673435691" watchObservedRunningTime="2026-09-10 00:01:27.424112218 +0000 UTC m=+767.675248261"
Sep 10 00:01:38.415797 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:38.415767    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-f5g58"
Sep 10 00:01:48.449218 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.449182    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"]
Sep 10 00:01:48.453851 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.453825    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.456387 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.456366    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 00:01:48.457189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.457171    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 00:01:48.457286 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.457186    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pbjt6\""
Sep 10 00:01:48.457286 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.457205    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 00:01:48.462273 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.462235    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"]
Sep 10 00:01:48.544498 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.544463    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5snf9\" (UniqueName: \"kubernetes.io/projected/498f2657-87b5-4e85-ac1b-464cacbaa819-kube-api-access-5snf9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.544641 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.544509    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/498f2657-87b5-4e85-ac1b-464cacbaa819-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.544641 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.544527    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.544747 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.544635    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.544747 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.544667    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.544747 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.544699    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.646028 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.645990    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.646028 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.646019    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.646249 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.646043    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.646249 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.646102    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5snf9\" (UniqueName: \"kubernetes.io/projected/498f2657-87b5-4e85-ac1b-464cacbaa819-kube-api-access-5snf9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.646249 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.646133    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/498f2657-87b5-4e85-ac1b-464cacbaa819-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.646249 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.646148    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.646461 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.646441    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.646515 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.646484    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.646596 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.646567    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.648476 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.648447    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/498f2657-87b5-4e85-ac1b-464cacbaa819-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.648607 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.648592    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/498f2657-87b5-4e85-ac1b-464cacbaa819-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.653851 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.653830    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5snf9\" (UniqueName: \"kubernetes.io/projected/498f2657-87b5-4e85-ac1b-464cacbaa819-kube-api-access-5snf9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-hb9d2\" (UID: \"498f2657-87b5-4e85-ac1b-464cacbaa819\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:48.765850 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:48.765824    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:49.034079 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:49.034042    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"]
Sep 10 00:01:49.035799 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:01:49.035775    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498f2657_87b5_4e85_ac1b_464cacbaa819.slice/crio-b0141ce8522e00ec94b6a51564aec765823c52e05d41dea861937f118946a362 WatchSource:0}: Error finding container b0141ce8522e00ec94b6a51564aec765823c52e05d41dea861937f118946a362: Status 404 returned error can't find the container with id b0141ce8522e00ec94b6a51564aec765823c52e05d41dea861937f118946a362
Sep 10 00:01:49.520708 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:49.520673    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2" event={"ID":"498f2657-87b5-4e85-ac1b-464cacbaa819","Type":"ContainerStarted","Data":"b0141ce8522e00ec94b6a51564aec765823c52e05d41dea861937f118946a362"}
Sep 10 00:01:50.526555 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:50.526519    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2" event={"ID":"498f2657-87b5-4e85-ac1b-464cacbaa819","Type":"ContainerStarted","Data":"bc8484b7ad12aefb10a497365406638c83c1f5b9b46a546dba0b412ac19c6d14"}
Sep 10 00:01:50.546855 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:50.546809    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2" podStartSLOduration=1.442474317 podStartE2EDuration="2.546795361s" totalImagesPullingTime="1.104321044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:01:50.545989479 +0000 UTC m=+790.797125522" watchObservedRunningTime="2026-09-10 00:01:50.546795361 +0000 UTC m=+790.797931404"
Sep 10 00:01:57.658635 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.658602    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"]
Sep 10 00:01:57.662195 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.662174    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.664444 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.664416    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 00:01:57.673076 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.673038    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"]
Sep 10 00:01:57.715048 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.715023    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.715173 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.715076    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.715173 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.715094    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.715173 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.715153    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.715322 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.715203    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dl6d\" (UniqueName: \"kubernetes.io/projected/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-kube-api-access-6dl6d\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.715322 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.715233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.815618 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.815592    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.815729 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.815634    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.815729 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.815655    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.815729 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.815680    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.815729 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.815711    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dl6d\" (UniqueName: \"kubernetes.io/projected/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-kube-api-access-6dl6d\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.815922 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.815738    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.816111 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.816083    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.816218 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.816147    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.816218 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.816146    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.817745 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.817718    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.818137 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.818118    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.823119 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.823102    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dl6d\" (UniqueName: \"kubernetes.io/projected/1e231bfd-72e2-46be-81c6-8ebc86d0c0a0-kube-api-access-6dl6d\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs\" (UID: \"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:57.972355 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:57.972331    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:01:58.240397 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:58.240368    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"]
Sep 10 00:01:58.241759 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:01:58.241735    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e231bfd_72e2_46be_81c6_8ebc86d0c0a0.slice/crio-7f8379f057da33ae7cb8e06d800015f698040bee8c0c2f5c2d4b67894fda711a WatchSource:0}: Error finding container 7f8379f057da33ae7cb8e06d800015f698040bee8c0c2f5c2d4b67894fda711a: Status 404 returned error can't find the container with id 7f8379f057da33ae7cb8e06d800015f698040bee8c0c2f5c2d4b67894fda711a
Sep 10 00:01:58.563081 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:58.562982    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs" event={"ID":"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0","Type":"ContainerStarted","Data":"1a13e0c52dc50477365110f6d4dc1882a5fe65d0c407c7b62528e25cb5e240b1"}
Sep 10 00:01:58.563081 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:58.563044    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs" event={"ID":"1e231bfd-72e2-46be-81c6-8ebc86d0c0a0","Type":"ContainerStarted","Data":"7f8379f057da33ae7cb8e06d800015f698040bee8c0c2f5c2d4b67894fda711a"}
Sep 10 00:01:58.587070 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:58.586995    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs" podStartSLOduration=1.421589177 podStartE2EDuration="1.586961659s" totalImagesPullingTime="165.372482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:01:58.57810318 +0000 UTC m=+798.829239224" watchObservedRunningTime="2026-09-10 00:01:58.586961659 +0000 UTC m=+798.838097746"
Sep 10 00:01:58.766300 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:58.766259    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:58.766667 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:58.766350    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:58.779643 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:58.779612    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:01:59.578348 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:01:59.578322    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-hb9d2"
Sep 10 00:02:07.973502 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:07.973420    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:02:07.973877 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:07.973537    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:02:07.985765 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:07.985738    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:02:08.615979 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:08.615947    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs"
Sep 10 00:02:22.655380 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.655333    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"]
Sep 10 00:02:22.661532 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.661509    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.663801 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.663783    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 00:02:22.666855 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.666828    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"]
Sep 10 00:02:22.820720 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.820682    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.820888 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.820731    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899caf63-d38d-4856-928e-f4f790c44bc6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.820888 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.820749    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896q5\" (UniqueName: \"kubernetes.io/projected/899caf63-d38d-4856-928e-f4f790c44bc6-kube-api-access-896q5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.820888 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.820855    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.821030 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.820914    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.821030 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.820940    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.921891 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.921818    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899caf63-d38d-4856-928e-f4f790c44bc6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.921891 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.921853    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896q5\" (UniqueName: \"kubernetes.io/projected/899caf63-d38d-4856-928e-f4f790c44bc6-kube-api-access-896q5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.921891 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.921877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.922184 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.921907    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.922184 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.921931    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.922184 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.922015    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.922457 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.922432    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.922564 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.922432    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.922564 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.922549    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.924236 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.924209    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/899caf63-d38d-4856-928e-f4f790c44bc6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.924506 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.924484    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/899caf63-d38d-4856-928e-f4f790c44bc6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.928946 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.928920    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-896q5\" (UniqueName: \"kubernetes.io/projected/899caf63-d38d-4856-928e-f4f790c44bc6-kube-api-access-896q5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj\" (UID: \"899caf63-d38d-4856-928e-f4f790c44bc6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:22.973468 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:22.973444    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:23.243696 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:23.243673    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"]
Sep 10 00:02:23.679682 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:23.679609    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj" event={"ID":"899caf63-d38d-4856-928e-f4f790c44bc6","Type":"ContainerStarted","Data":"74693a8e349f639b3684f73621a8b41240fabd9aa792663179f2174a9692fa4e"}
Sep 10 00:02:23.679682 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:23.679667    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj" event={"ID":"899caf63-d38d-4856-928e-f4f790c44bc6","Type":"ContainerStarted","Data":"186f5e4a48f7ec3f08ad28ecb0fd9c7e104dc60b48c1db1fc1f04c317de58f67"}
Sep 10 00:02:23.698069 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:23.698003    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj" podStartSLOduration=1.521591615 podStartE2EDuration="1.697989669s" totalImagesPullingTime="176.398054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:02:23.695741715 +0000 UTC m=+823.946877759" watchObservedRunningTime="2026-09-10 00:02:23.697989669 +0000 UTC m=+823.949125713"
Sep 10 00:02:28.455842 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.455801    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"]
Sep 10 00:02:28.459895 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.459873    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.462329 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.462306    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 00:02:28.470148 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.470126    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"]
Sep 10 00:02:28.572640 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.572610    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.572640 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.572639    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.572784 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.572673    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9571ce8-a6ad-4d01-8744-439040e11b6c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.572784 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.572743    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.572784 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.572771    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.572882 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.572827    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86wm8\" (UniqueName: \"kubernetes.io/projected/e9571ce8-a6ad-4d01-8744-439040e11b6c-kube-api-access-86wm8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.673439 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.673405    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.673565 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.673443    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.673565 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.673477    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9571ce8-a6ad-4d01-8744-439040e11b6c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.673565 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.673501    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.673565 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.673542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.673772 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.673626    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86wm8\" (UniqueName: \"kubernetes.io/projected/e9571ce8-a6ad-4d01-8744-439040e11b6c-kube-api-access-86wm8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.673852 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.673831    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.673924 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.673904    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.673993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.673972    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.675703 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.675676    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9571ce8-a6ad-4d01-8744-439040e11b6c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.675873 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.675854    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9571ce8-a6ad-4d01-8744-439040e11b6c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.681001 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.680980    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86wm8\" (UniqueName: \"kubernetes.io/projected/e9571ce8-a6ad-4d01-8744-439040e11b6c-kube-api-access-86wm8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wb9s4\" (UID: \"e9571ce8-a6ad-4d01-8744-439040e11b6c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:28.773619 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:28.773571    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:29.043625 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:29.043599    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"]
Sep 10 00:02:29.045823 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:02:29.045793    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9571ce8_a6ad_4d01_8744_439040e11b6c.slice/crio-a1b9c3e9b125f618b6820cc81f1e5bacb8c02a363a8bd99f2eda35469b99109d WatchSource:0}: Error finding container a1b9c3e9b125f618b6820cc81f1e5bacb8c02a363a8bd99f2eda35469b99109d: Status 404 returned error can't find the container with id a1b9c3e9b125f618b6820cc81f1e5bacb8c02a363a8bd99f2eda35469b99109d
Sep 10 00:02:29.710701 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:29.710669    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4" event={"ID":"e9571ce8-a6ad-4d01-8744-439040e11b6c","Type":"ContainerStarted","Data":"80d40a0fa8435dddcd422f8b504d5695ab2dc557c7eff9ede1f442732e8c203b"}
Sep 10 00:02:29.711073 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:29.710724    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4" event={"ID":"e9571ce8-a6ad-4d01-8744-439040e11b6c","Type":"ContainerStarted","Data":"a1b9c3e9b125f618b6820cc81f1e5bacb8c02a363a8bd99f2eda35469b99109d"}
Sep 10 00:02:29.728575 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:29.728529    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4" podStartSLOduration=1.531050156 podStartE2EDuration="1.728514833s" totalImagesPullingTime="197.464677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:02:29.726282559 +0000 UTC m=+829.977418602" watchObservedRunningTime="2026-09-10 00:02:29.728514833 +0000 UTC m=+829.979650877"
Sep 10 00:02:32.974594 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:32.974543    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:32.974594 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:32.974618    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:32.987437 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:32.987404    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:33.742164 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:33.742130    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj"
Sep 10 00:02:38.773891 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:38.773850    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:38.774376 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:38.773939    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:38.787155 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:38.787133    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:02:39.777328 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:02:39.777297    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wb9s4"
Sep 10 00:03:41.499504 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:41.499471    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-658dbc9d99-t9lnr"]
Sep 10 00:03:41.503075 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:41.503042    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-658dbc9d99-t9lnr"
Sep 10 00:03:41.510868 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:41.510833    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-658dbc9d99-t9lnr"]
Sep 10 00:03:41.569624 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:41.569589    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4675932e-4a62-4127-a44d-d2d9d2bbc8f3-tls-cert\") pod \"authorino-658dbc9d99-t9lnr\" (UID: \"4675932e-4a62-4127-a44d-d2d9d2bbc8f3\") " pod="kuadrant-system/authorino-658dbc9d99-t9lnr"
Sep 10 00:03:41.569624 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:41.569625    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72h4\" (UniqueName: \"kubernetes.io/projected/4675932e-4a62-4127-a44d-d2d9d2bbc8f3-kube-api-access-w72h4\") pod \"authorino-658dbc9d99-t9lnr\" (UID: \"4675932e-4a62-4127-a44d-d2d9d2bbc8f3\") " pod="kuadrant-system/authorino-658dbc9d99-t9lnr"
Sep 10 00:03:41.671246 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:41.671211    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4675932e-4a62-4127-a44d-d2d9d2bbc8f3-tls-cert\") pod \"authorino-658dbc9d99-t9lnr\" (UID: \"4675932e-4a62-4127-a44d-d2d9d2bbc8f3\") " pod="kuadrant-system/authorino-658dbc9d99-t9lnr"
Sep 10 00:03:41.671246 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:41.671247    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72h4\" (UniqueName: \"kubernetes.io/projected/4675932e-4a62-4127-a44d-d2d9d2bbc8f3-kube-api-access-w72h4\") pod \"authorino-658dbc9d99-t9lnr\" (UID: \"4675932e-4a62-4127-a44d-d2d9d2bbc8f3\") " pod="kuadrant-system/authorino-658dbc9d99-t9lnr"
Sep 10 00:03:41.673609 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:41.673579    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4675932e-4a62-4127-a44d-d2d9d2bbc8f3-tls-cert\") pod \"authorino-658dbc9d99-t9lnr\" (UID: \"4675932e-4a62-4127-a44d-d2d9d2bbc8f3\") " pod="kuadrant-system/authorino-658dbc9d99-t9lnr"
Sep 10 00:03:41.679316 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:41.679286    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72h4\" (UniqueName: \"kubernetes.io/projected/4675932e-4a62-4127-a44d-d2d9d2bbc8f3-kube-api-access-w72h4\") pod \"authorino-658dbc9d99-t9lnr\" (UID: \"4675932e-4a62-4127-a44d-d2d9d2bbc8f3\") " pod="kuadrant-system/authorino-658dbc9d99-t9lnr"
Sep 10 00:03:41.814041 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:41.813963    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-658dbc9d99-t9lnr"
Sep 10 00:03:42.080629 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:42.080607    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-658dbc9d99-t9lnr"]
Sep 10 00:03:42.082550 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:03:42.082518    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4675932e_4a62_4127_a44d_d2d9d2bbc8f3.slice/crio-228b3978a815fcdd2be95942978fed4810adadf14ffb16684d58a080a6453ec9 WatchSource:0}: Error finding container 228b3978a815fcdd2be95942978fed4810adadf14ffb16684d58a080a6453ec9: Status 404 returned error can't find the container with id 228b3978a815fcdd2be95942978fed4810adadf14ffb16684d58a080a6453ec9
Sep 10 00:03:43.072951 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:43.072918    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-658dbc9d99-t9lnr" event={"ID":"4675932e-4a62-4127-a44d-d2d9d2bbc8f3","Type":"ContainerStarted","Data":"228b3978a815fcdd2be95942978fed4810adadf14ffb16684d58a080a6453ec9"}
Sep 10 00:03:44.083404 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:44.083356    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-658dbc9d99-t9lnr" event={"ID":"4675932e-4a62-4127-a44d-d2d9d2bbc8f3","Type":"ContainerStarted","Data":"7a3d545a02ead1e4957e32a2ccf230fda2458b32a30e5d9e5679e7c949a1974e"}
Sep 10 00:03:44.107570 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:44.107498    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-658dbc9d99-t9lnr" podStartSLOduration=1.505064697 podStartE2EDuration="3.107476879s" totalImagesPullingTime="1.602412182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:03:44.1069391 +0000 UTC m=+904.358075143" watchObservedRunningTime="2026-09-10 00:03:44.107476879 +0000 UTC m=+904.358612924"
Sep 10 00:03:44.133852 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:44.133817    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fd7564778-28jfc"]
Sep 10 00:03:44.134126 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:44.134085    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-fd7564778-28jfc" podUID="d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0" containerName="authorino" containerID="cri-o://5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c" gracePeriod=30
Sep 10 00:03:44.399684 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:44.399657    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd7564778-28jfc"
Sep 10 00:03:44.495086 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:44.494499    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-tls-cert\") pod \"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0\" (UID: \"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0\") "
Sep 10 00:03:44.495086 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:44.494588    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n6h5\" (UniqueName: \"kubernetes.io/projected/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-kube-api-access-7n6h5\") pod \"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0\" (UID: \"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0\") "
Sep 10 00:03:44.499003 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:44.498963    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-kube-api-access-7n6h5" (OuterVolumeSpecName: "kube-api-access-7n6h5") pod "d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0" (UID: "d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0"). InnerVolumeSpecName "kube-api-access-7n6h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:03:44.507557 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:44.507528    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0" (UID: "d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:03:44.595961 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:44.595917    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n6h5\" (UniqueName: \"kubernetes.io/projected/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-kube-api-access-7n6h5\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:03:44.595961 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:44.595950    2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0-tls-cert\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:03:45.090146 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:45.090113    2556 generic.go:356] "Generic (PLEG): container finished" podID="d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0" containerID="5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c" exitCode=0
Sep 10 00:03:45.090546 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:45.090173    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-fd7564778-28jfc"
Sep 10 00:03:45.090546 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:45.090200    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd7564778-28jfc" event={"ID":"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0","Type":"ContainerDied","Data":"5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c"}
Sep 10 00:03:45.090546 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:45.090243    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-fd7564778-28jfc" event={"ID":"d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0","Type":"ContainerDied","Data":"29f5f88a67793eed0b5e60d671b5ce55a43cf1318acd7a59b571ea747c36a9a1"}
Sep 10 00:03:45.090546 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:45.090260    2556 scope.go:117] "RemoveContainer" containerID="5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c"
Sep 10 00:03:45.099285 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:45.099269    2556 scope.go:117] "RemoveContainer" containerID="5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c"
Sep 10 00:03:45.099549 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:03:45.099528    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c\": container with ID starting with 5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c not found: ID does not exist" containerID="5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c"
Sep 10 00:03:45.099600 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:45.099556    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c"} err="failed to get container status \"5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c\": rpc error: code = NotFound desc = could not find container \"5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c\": container with ID starting with 5f1ce82e11dddf7da1ff89e826899da4815e43cc2dbf60e9979db394955a703c not found: ID does not exist"
Sep 10 00:03:45.116828 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:45.116796    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-fd7564778-28jfc"]
Sep 10 00:03:45.119185 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:45.119165    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-fd7564778-28jfc"]
Sep 10 00:03:46.269102 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:03:46.269040    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0" path="/var/lib/kubelet/pods/d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0/volumes"
Sep 10 00:04:03.463660 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:03.463621    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7801-2e0267"]
Sep 10 00:04:03.464133 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:03.464014    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0" containerName="authorino"
Sep 10 00:04:03.464133 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:03.464025    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0" containerName="authorino"
Sep 10 00:04:03.464133 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:03.464099    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1c0e6d5-fe74-4a44-b9b4-e906f17e04e0" containerName="authorino"
Sep 10 00:04:03.467049 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:03.467032    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7801-2e0267"
Sep 10 00:04:03.469302 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:03.469278    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gtwx5\""
Sep 10 00:04:03.472676 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:03.472655    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7801-2e0267"]
Sep 10 00:04:03.574396 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:03.574362    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt54p\" (UniqueName: \"kubernetes.io/projected/ca2aba45-e0a4-49d7-8294-798f2bd3178f-kube-api-access-lt54p\") pod \"test-curl-7801-2e0267\" (UID: \"ca2aba45-e0a4-49d7-8294-798f2bd3178f\") " pod="opendatahub/test-curl-7801-2e0267"
Sep 10 00:04:03.675771 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:03.675736    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt54p\" (UniqueName: \"kubernetes.io/projected/ca2aba45-e0a4-49d7-8294-798f2bd3178f-kube-api-access-lt54p\") pod \"test-curl-7801-2e0267\" (UID: \"ca2aba45-e0a4-49d7-8294-798f2bd3178f\") " pod="opendatahub/test-curl-7801-2e0267"
Sep 10 00:04:03.687371 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:03.687341    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt54p\" (UniqueName: \"kubernetes.io/projected/ca2aba45-e0a4-49d7-8294-798f2bd3178f-kube-api-access-lt54p\") pod \"test-curl-7801-2e0267\" (UID: \"ca2aba45-e0a4-49d7-8294-798f2bd3178f\") " pod="opendatahub/test-curl-7801-2e0267"
Sep 10 00:04:03.777279 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:03.777208    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7801-2e0267"
Sep 10 00:04:04.044815 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:04.044784    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7801-2e0267"]
Sep 10 00:04:04.046875 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:04.046850    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca2aba45_e0a4_49d7_8294_798f2bd3178f.slice/crio-b7b752a83ef8158cf4639d2f7b70f292331947160f71e717ace4520b1b7432a3 WatchSource:0}: Error finding container b7b752a83ef8158cf4639d2f7b70f292331947160f71e717ace4520b1b7432a3: Status 404 returned error can't find the container with id b7b752a83ef8158cf4639d2f7b70f292331947160f71e717ace4520b1b7432a3
Sep 10 00:04:04.048874 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:04.048852    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 00:04:04.188671 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:04.188638    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7801-2e0267" event={"ID":"ca2aba45-e0a4-49d7-8294-798f2bd3178f","Type":"ContainerStarted","Data":"b7b752a83ef8158cf4639d2f7b70f292331947160f71e717ace4520b1b7432a3"}
Sep 10 00:04:07.174548 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.174512    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"]
Sep 10 00:04:07.178150 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.178129    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:04:07.180956 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.180930    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f0f965-dockercfg-khb6t\""
Sep 10 00:04:07.181091 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.180932    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f0f965\""
Sep 10 00:04:07.188015 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.187993    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"]
Sep 10 00:04:07.311933 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.311900    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e0c580-2c3b-4227-b870-870eada28626-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66\" (UID: \"72e0c580-2c3b-4227-b870-870eada28626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:04:07.312120 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.311949    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwg9t\" (UniqueName: \"kubernetes.io/projected/72e0c580-2c3b-4227-b870-870eada28626-kube-api-access-pwg9t\") pod \"payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66\" (UID: \"72e0c580-2c3b-4227-b870-870eada28626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:04:07.412466 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.412435    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e0c580-2c3b-4227-b870-870eada28626-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66\" (UID: \"72e0c580-2c3b-4227-b870-870eada28626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:04:07.412621 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.412506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwg9t\" (UniqueName: \"kubernetes.io/projected/72e0c580-2c3b-4227-b870-870eada28626-kube-api-access-pwg9t\") pod \"payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66\" (UID: \"72e0c580-2c3b-4227-b870-870eada28626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:04:07.413159 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.413140    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e0c580-2c3b-4227-b870-870eada28626-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66\" (UID: \"72e0c580-2c3b-4227-b870-870eada28626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:04:07.424424 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.424390    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwg9t\" (UniqueName: \"kubernetes.io/projected/72e0c580-2c3b-4227-b870-870eada28626-kube-api-access-pwg9t\") pod \"payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66\" (UID: \"72e0c580-2c3b-4227-b870-870eada28626\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:04:07.490040 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.490013    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:04:07.762035 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:07.761956    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"]
Sep 10 00:04:07.764656 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:07.764628    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e0c580_2c3b_4227_b870_870eada28626.slice/crio-ff4bbd81b468837291bd8bbf818595a5bebcc889ff5115fa1ccf2b407ff30ecb WatchSource:0}: Error finding container ff4bbd81b468837291bd8bbf818595a5bebcc889ff5115fa1ccf2b407ff30ecb: Status 404 returned error can't find the container with id ff4bbd81b468837291bd8bbf818595a5bebcc889ff5115fa1ccf2b407ff30ecb
Sep 10 00:04:08.131989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.131915    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"]
Sep 10 00:04:08.135260 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.135244    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:04:08.138227 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.138205    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-01b7d5\""
Sep 10 00:04:08.138338 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.138292    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-01b7d5-dockercfg-4brf8\""
Sep 10 00:04:08.148514 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.148487    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"]
Sep 10 00:04:08.162568 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.162535    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"]
Sep 10 00:04:08.166367 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.166343    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:04:08.174328 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.174304    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"]
Sep 10 00:04:08.210647 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.210617    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7801-2e0267" event={"ID":"ca2aba45-e0a4-49d7-8294-798f2bd3178f","Type":"ContainerStarted","Data":"5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c"}
Sep 10 00:04:08.212267 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.212244    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66" event={"ID":"72e0c580-2c3b-4227-b870-870eada28626","Type":"ContainerStarted","Data":"d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e"}
Sep 10 00:04:08.212377 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.212285    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66" event={"ID":"72e0c580-2c3b-4227-b870-870eada28626","Type":"ContainerStarted","Data":"ff4bbd81b468837291bd8bbf818595a5bebcc889ff5115fa1ccf2b407ff30ecb"}
Sep 10 00:04:08.212377 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.212315    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:04:08.223699 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.223657    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7801-2e0267" podStartSLOduration=1.789850945 podStartE2EDuration="5.223642184s" totalImagesPullingTime="3.433791239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:08.223643079 +0000 UTC m=+928.474779121" watchObservedRunningTime="2026-09-10 00:04:08.223642184 +0000 UTC m=+928.474778226"
Sep 10 00:04:08.245141 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.245088    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66" podStartSLOduration=1.245071579 podStartE2EDuration="1.245071579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:08.242535709 +0000 UTC m=+928.493671780" watchObservedRunningTime="2026-09-10 00:04:08.245071579 +0000 UTC m=+928.496207620"
Sep 10 00:04:08.321096 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.321044    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06d7cda-d92c-4ad0-83d3-34177df92834-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl\" (UID: \"a06d7cda-d92c-4ad0-83d3-34177df92834\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:04:08.321096 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.321101    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhlr6\" (UniqueName: \"kubernetes.io/projected/a06d7cda-d92c-4ad0-83d3-34177df92834-kube-api-access-vhlr6\") pod \"payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl\" (UID: \"a06d7cda-d92c-4ad0-83d3-34177df92834\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:04:08.321348 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.321127    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmrg\" (UniqueName: \"kubernetes.io/projected/df6bb0d0-df23-4a5e-ae76-13482129b996-kube-api-access-slmrg\") pod \"payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw\" (UID: \"df6bb0d0-df23-4a5e-ae76-13482129b996\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:04:08.321348 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.321252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df6bb0d0-df23-4a5e-ae76-13482129b996-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw\" (UID: \"df6bb0d0-df23-4a5e-ae76-13482129b996\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:04:08.422539 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.422464    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df6bb0d0-df23-4a5e-ae76-13482129b996-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw\" (UID: \"df6bb0d0-df23-4a5e-ae76-13482129b996\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:04:08.422539 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.422530    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06d7cda-d92c-4ad0-83d3-34177df92834-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl\" (UID: \"a06d7cda-d92c-4ad0-83d3-34177df92834\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:04:08.422732 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.422546    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhlr6\" (UniqueName: \"kubernetes.io/projected/a06d7cda-d92c-4ad0-83d3-34177df92834-kube-api-access-vhlr6\") pod \"payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl\" (UID: \"a06d7cda-d92c-4ad0-83d3-34177df92834\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:04:08.422732 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.422563    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmrg\" (UniqueName: \"kubernetes.io/projected/df6bb0d0-df23-4a5e-ae76-13482129b996-kube-api-access-slmrg\") pod \"payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw\" (UID: \"df6bb0d0-df23-4a5e-ae76-13482129b996\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:04:08.423456 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.423430    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df6bb0d0-df23-4a5e-ae76-13482129b996-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw\" (UID: \"df6bb0d0-df23-4a5e-ae76-13482129b996\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:04:08.423539 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.423430    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06d7cda-d92c-4ad0-83d3-34177df92834-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl\" (UID: \"a06d7cda-d92c-4ad0-83d3-34177df92834\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:04:08.430146 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.430122    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhlr6\" (UniqueName: \"kubernetes.io/projected/a06d7cda-d92c-4ad0-83d3-34177df92834-kube-api-access-vhlr6\") pod \"payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl\" (UID: \"a06d7cda-d92c-4ad0-83d3-34177df92834\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:04:08.430275 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.430245    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmrg\" (UniqueName: \"kubernetes.io/projected/df6bb0d0-df23-4a5e-ae76-13482129b996-kube-api-access-slmrg\") pod \"payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw\" (UID: \"df6bb0d0-df23-4a5e-ae76-13482129b996\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:04:08.447267 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.447243    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:04:08.478639 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.478614    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:04:08.727494 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.727462    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"]
Sep 10 00:04:08.728960 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:08.728934    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06d7cda_d92c_4ad0_83d3_34177df92834.slice/crio-56a39db71d4baad5f3d3e00fafe8800260e6f02fe3396df8094f48b8126a6b3a WatchSource:0}: Error finding container 56a39db71d4baad5f3d3e00fafe8800260e6f02fe3396df8094f48b8126a6b3a: Status 404 returned error can't find the container with id 56a39db71d4baad5f3d3e00fafe8800260e6f02fe3396df8094f48b8126a6b3a
Sep 10 00:04:08.761155 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.761128    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"]
Sep 10 00:04:08.762769 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:08.762743    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6bb0d0_df23_4a5e_ae76_13482129b996.slice/crio-67fed14a7d2a0d18183672198d253b324f279223e40fee7dcd1e5c1192d0eda2 WatchSource:0}: Error finding container 67fed14a7d2a0d18183672198d253b324f279223e40fee7dcd1e5c1192d0eda2: Status 404 returned error can't find the container with id 67fed14a7d2a0d18183672198d253b324f279223e40fee7dcd1e5c1192d0eda2
Sep 10 00:04:08.796451 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.796429    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7801-2e0267"]
Sep 10 00:04:08.801078 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:08.801002    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7801-2e0267"]
Sep 10 00:04:09.199792 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.199713    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"]
Sep 10 00:04:09.200145 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.200132    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2aba45-e0a4-49d7-8294-798f2bd3178f" containerName="test-curl-7801-2e0267"
Sep 10 00:04:09.200194 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.200146    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2aba45-e0a4-49d7-8294-798f2bd3178f" containerName="test-curl-7801-2e0267"
Sep 10 00:04:09.200246 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.200237    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2aba45-e0a4-49d7-8294-798f2bd3178f" containerName="test-curl-7801-2e0267"
Sep 10 00:04:09.203517 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.203494    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:04:09.206043 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.206017    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-cc0040\""
Sep 10 00:04:09.206198 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.206100    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-cc0040-dockercfg-4ct2t\""
Sep 10 00:04:09.213904 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.213881    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"]
Sep 10 00:04:09.219941 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.219911    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw" event={"ID":"df6bb0d0-df23-4a5e-ae76-13482129b996","Type":"ContainerStarted","Data":"08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e"}
Sep 10 00:04:09.220076 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.219960    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw" event={"ID":"df6bb0d0-df23-4a5e-ae76-13482129b996","Type":"ContainerStarted","Data":"67fed14a7d2a0d18183672198d253b324f279223e40fee7dcd1e5c1192d0eda2"}
Sep 10 00:04:09.220076 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.219992    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:04:09.221936 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.221909    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl" event={"ID":"a06d7cda-d92c-4ad0-83d3-34177df92834","Type":"ContainerStarted","Data":"2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092"}
Sep 10 00:04:09.222031 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.221961    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl" event={"ID":"a06d7cda-d92c-4ad0-83d3-34177df92834","Type":"ContainerStarted","Data":"56a39db71d4baad5f3d3e00fafe8800260e6f02fe3396df8094f48b8126a6b3a"}
Sep 10 00:04:09.222326 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:04:09.222295    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7801-2e0267" container="test-curl-7801-2e0267"
Sep 10 00:04:09.222455 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.222437    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:04:09.240130 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.240088    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw" podStartSLOduration=1.240074307 podStartE2EDuration="1.240074307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:09.237821697 +0000 UTC m=+929.488957743" watchObservedRunningTime="2026-09-10 00:04:09.240074307 +0000 UTC m=+929.491210347"
Sep 10 00:04:09.251266 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.251230    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"]
Sep 10 00:04:09.254897 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.254872    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:04:09.255105 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.255039    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl" podStartSLOduration=1.255023847 podStartE2EDuration="1.255023847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:09.254306248 +0000 UTC m=+929.505442294" watchObservedRunningTime="2026-09-10 00:04:09.255023847 +0000 UTC m=+929.506159891"
Sep 10 00:04:09.263452 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.262905    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"]
Sep 10 00:04:09.329862 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.329829    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27\" (UID: \"7afd4e08-7c7d-4525-88b4-2aaa32c171b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:04:09.330047 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.329869    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rj8\" (UniqueName: \"kubernetes.io/projected/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-kube-api-access-l4rj8\") pod \"payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27\" (UID: \"7afd4e08-7c7d-4525-88b4-2aaa32c171b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:04:09.430464 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.430431    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cr2z\" (UniqueName: \"kubernetes.io/projected/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-kube-api-access-8cr2z\") pod \"payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8\" (UID: \"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:04:09.430637 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.430473    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27\" (UID: \"7afd4e08-7c7d-4525-88b4-2aaa32c171b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:04:09.430637 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.430491    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rj8\" (UniqueName: \"kubernetes.io/projected/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-kube-api-access-l4rj8\") pod \"payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27\" (UID: \"7afd4e08-7c7d-4525-88b4-2aaa32c171b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:04:09.430637 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.430553    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8\" (UID: \"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:04:09.431226 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.431202    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27\" (UID: \"7afd4e08-7c7d-4525-88b4-2aaa32c171b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:04:09.438766 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.438741    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rj8\" (UniqueName: \"kubernetes.io/projected/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-kube-api-access-l4rj8\") pod \"payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27\" (UID: \"7afd4e08-7c7d-4525-88b4-2aaa32c171b9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:04:09.515324 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.515293    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:04:09.531624 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.531590    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cr2z\" (UniqueName: \"kubernetes.io/projected/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-kube-api-access-8cr2z\") pod \"payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8\" (UID: \"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:04:09.531790 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.531682    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8\" (UID: \"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:04:09.532379 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.532351    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8\" (UID: \"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:04:09.539261 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.539237    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cr2z\" (UniqueName: \"kubernetes.io/projected/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-kube-api-access-8cr2z\") pod \"payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8\" (UID: \"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:04:09.567989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.567751    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:04:09.788791 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.788761    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"]
Sep 10 00:04:09.791233 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:09.791206    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7afd4e08_7c7d_4525_88b4_2aaa32c171b9.slice/crio-36098ce1b46a764ec99ebab7e9cb00a37537e7eb16053bd0def9f80c8a607471 WatchSource:0}: Error finding container 36098ce1b46a764ec99ebab7e9cb00a37537e7eb16053bd0def9f80c8a607471: Status 404 returned error can't find the container with id 36098ce1b46a764ec99ebab7e9cb00a37537e7eb16053bd0def9f80c8a607471
Sep 10 00:04:09.859125 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:09.859096    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"]
Sep 10 00:04:09.861156 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:09.861116    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f14d8b_d9e0_4c21_8638_4cab4e6ad9d7.slice/crio-b2183fef4170ff0b99f9e22bb7a70f6ba7e59474b72e07f32f0b27b2b2ff0c7b WatchSource:0}: Error finding container b2183fef4170ff0b99f9e22bb7a70f6ba7e59474b72e07f32f0b27b2b2ff0c7b: Status 404 returned error can't find the container with id b2183fef4170ff0b99f9e22bb7a70f6ba7e59474b72e07f32f0b27b2b2ff0c7b
Sep 10 00:04:10.228679 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:10.228642    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8" event={"ID":"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7","Type":"ContainerStarted","Data":"ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269"}
Sep 10 00:04:10.229177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:10.228706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8" event={"ID":"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7","Type":"ContainerStarted","Data":"b2183fef4170ff0b99f9e22bb7a70f6ba7e59474b72e07f32f0b27b2b2ff0c7b"}
Sep 10 00:04:10.229177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:10.228736    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:04:10.230384 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:10.230349    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27" event={"ID":"7afd4e08-7c7d-4525-88b4-2aaa32c171b9","Type":"ContainerStarted","Data":"841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325"}
Sep 10 00:04:10.230490 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:10.230391    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27" event={"ID":"7afd4e08-7c7d-4525-88b4-2aaa32c171b9","Type":"ContainerStarted","Data":"36098ce1b46a764ec99ebab7e9cb00a37537e7eb16053bd0def9f80c8a607471"}
Sep 10 00:04:10.230798 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:10.230770    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7801-2e0267" podUID="ca2aba45-e0a4-49d7-8294-798f2bd3178f" containerName="test-curl-7801-2e0267" containerID="cri-o://5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c" gracePeriod=2
Sep 10 00:04:10.248875 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:10.248812    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8" podStartSLOduration=1.248791105 podStartE2EDuration="1.248791105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:10.245995925 +0000 UTC m=+930.497131973" watchObservedRunningTime="2026-09-10 00:04:10.248791105 +0000 UTC m=+930.499927155"
Sep 10 00:04:10.265599 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:10.265540    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27" podStartSLOduration=1.265520817 podStartE2EDuration="1.265520817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:10.264432329 +0000 UTC m=+930.515568375" watchObservedRunningTime="2026-09-10 00:04:10.265520817 +0000 UTC m=+930.516656865"
Sep 10 00:04:10.266568 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:04:10.266540    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7801-2e0267\" is forbidden: User \"system:node:ip-10-0-139-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-211.ec2.internal' and this object" podUID="ca2aba45-e0a4-49d7-8294-798f2bd3178f" pod="opendatahub/test-curl-7801-2e0267"
Sep 10 00:04:10.276447 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:04:10.276413    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7801-2e0267\" is forbidden: User \"system:node:ip-10-0-139-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-211.ec2.internal' and this object" podUID="ca2aba45-e0a4-49d7-8294-798f2bd3178f" pod="opendatahub/test-curl-7801-2e0267"
Sep 10 00:04:11.235803 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:11.235767    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:04:12.390091 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:12.390049    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7801-2e0267"
Sep 10 00:04:12.559704 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:12.559613    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt54p\" (UniqueName: \"kubernetes.io/projected/ca2aba45-e0a4-49d7-8294-798f2bd3178f-kube-api-access-lt54p\") pod \"ca2aba45-e0a4-49d7-8294-798f2bd3178f\" (UID: \"ca2aba45-e0a4-49d7-8294-798f2bd3178f\") "
Sep 10 00:04:12.562118 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:12.561970    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2aba45-e0a4-49d7-8294-798f2bd3178f-kube-api-access-lt54p" (OuterVolumeSpecName: "kube-api-access-lt54p") pod "ca2aba45-e0a4-49d7-8294-798f2bd3178f" (UID: "ca2aba45-e0a4-49d7-8294-798f2bd3178f"). InnerVolumeSpecName "kube-api-access-lt54p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:04:12.660639 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:12.660591    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt54p\" (UniqueName: \"kubernetes.io/projected/ca2aba45-e0a4-49d7-8294-798f2bd3178f-kube-api-access-lt54p\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:04:13.246799 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:13.246758    2556 generic.go:356] "Generic (PLEG): container finished" podID="ca2aba45-e0a4-49d7-8294-798f2bd3178f" containerID="5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c" exitCode=137
Sep 10 00:04:13.246947 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:13.246806    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7801-2e0267"
Sep 10 00:04:13.246947 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:13.246847    2556 scope.go:117] "RemoveContainer" containerID="5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c"
Sep 10 00:04:13.255844 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:13.255820    2556 scope.go:117] "RemoveContainer" containerID="5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c"
Sep 10 00:04:13.256143 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:04:13.256124    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c\": container with ID starting with 5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c not found: ID does not exist" containerID="5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c"
Sep 10 00:04:13.256201 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:13.256153    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c"} err="failed to get container status \"5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c\": rpc error: code = NotFound desc = could not find container \"5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c\": container with ID starting with 5f27e0f792a869d809f15cecd9ae5ed74db0aa42daf1eea85d9294cf9300d37c not found: ID does not exist"
Sep 10 00:04:13.257396 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:04:13.257372    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7801-2e0267\" is forbidden: User \"system:node:ip-10-0-139-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-211.ec2.internal' and this object" podUID="ca2aba45-e0a4-49d7-8294-798f2bd3178f" pod="opendatahub/test-curl-7801-2e0267"
Sep 10 00:04:14.270000 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.269964    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca2aba45-e0a4-49d7-8294-798f2bd3178f" path="/var/lib/kubelet/pods/ca2aba45-e0a4-49d7-8294-798f2bd3178f/volumes"
Sep 10 00:04:14.517798 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.517761    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"]
Sep 10 00:04:14.522724 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.522665    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:14.525261 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.525237    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-46484ed0-dockercfg-d4pfs\""
Sep 10 00:04:14.525391 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.525237    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-46484ed0\""
Sep 10 00:04:14.530714 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.530462    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"]
Sep 10 00:04:14.680331 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.680291    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w\" (UID: \"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:14.680331 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.680331    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwfbq\" (UniqueName: \"kubernetes.io/projected/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-kube-api-access-jwfbq\") pod \"payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w\" (UID: \"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:14.780968 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.780881    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w\" (UID: \"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:14.780968 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.780918    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwfbq\" (UniqueName: \"kubernetes.io/projected/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-kube-api-access-jwfbq\") pod \"payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w\" (UID: \"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:14.781573 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.781548    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w\" (UID: \"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:14.794716 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.794689    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwfbq\" (UniqueName: \"kubernetes.io/projected/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-kube-api-access-jwfbq\") pod \"payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w\" (UID: \"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:14.835368 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:14.835341    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:15.107046 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:15.106960    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"]
Sep 10 00:04:15.110862 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:15.110837    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ddbbb2_69ed_4dc2_9f0f_98ec8e07be0b.slice/crio-7590e8cf00ee369b2407bdd6408da24ce256a10955768e04c5b3a424b84ed096 WatchSource:0}: Error finding container 7590e8cf00ee369b2407bdd6408da24ce256a10955768e04c5b3a424b84ed096: Status 404 returned error can't find the container with id 7590e8cf00ee369b2407bdd6408da24ce256a10955768e04c5b3a424b84ed096
Sep 10 00:04:15.260580 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:15.260544    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w" event={"ID":"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b","Type":"ContainerStarted","Data":"09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e"}
Sep 10 00:04:15.260759 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:15.260635    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w" event={"ID":"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b","Type":"ContainerStarted","Data":"7590e8cf00ee369b2407bdd6408da24ce256a10955768e04c5b3a424b84ed096"}
Sep 10 00:04:15.260759 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:15.260660    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:15.277319 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:15.277261    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w" podStartSLOduration=1.277242898 podStartE2EDuration="1.277242898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:15.275389413 +0000 UTC m=+935.526525458" watchObservedRunningTime="2026-09-10 00:04:15.277242898 +0000 UTC m=+935.528378940"
Sep 10 00:04:18.928313 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:18.928276    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"]
Sep 10 00:04:18.932200 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:18.932182    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:04:18.934766 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:18.934736    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-39c7cb\""
Sep 10 00:04:18.934906 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:18.934774    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-39c7cb-dockercfg-xpnqj\""
Sep 10 00:04:18.944980 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:18.944952    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"]
Sep 10 00:04:18.961045 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:18.961015    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"]
Sep 10 00:04:18.964687 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:18.964666    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:04:18.973256 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:18.973228    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"]
Sep 10 00:04:19.021544 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.021514    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxkmm\" (UniqueName: \"kubernetes.io/projected/c0fc24d3-92d2-4629-8652-776f05907e9c-kube-api-access-vxkmm\") pod \"payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck\" (UID: \"c0fc24d3-92d2-4629-8652-776f05907e9c\") " pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:04:19.021725 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.021567    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fc24d3-92d2-4629-8652-776f05907e9c-plugins-config-volume\") pod \"payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck\" (UID: \"c0fc24d3-92d2-4629-8652-776f05907e9c\") " pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:04:19.021725 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.021652    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4jfn\" (UniqueName: \"kubernetes.io/projected/8695216c-8df4-4b31-8bd4-11c2567e759f-kube-api-access-s4jfn\") pod \"payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx\" (UID: \"8695216c-8df4-4b31-8bd4-11c2567e759f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:04:19.021725 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.021692    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8695216c-8df4-4b31-8bd4-11c2567e759f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx\" (UID: \"8695216c-8df4-4b31-8bd4-11c2567e759f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:04:19.123235 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.123198    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxkmm\" (UniqueName: \"kubernetes.io/projected/c0fc24d3-92d2-4629-8652-776f05907e9c-kube-api-access-vxkmm\") pod \"payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck\" (UID: \"c0fc24d3-92d2-4629-8652-776f05907e9c\") " pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:04:19.123569 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.123537    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fc24d3-92d2-4629-8652-776f05907e9c-plugins-config-volume\") pod \"payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck\" (UID: \"c0fc24d3-92d2-4629-8652-776f05907e9c\") " pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:04:19.123780 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.123761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4jfn\" (UniqueName: \"kubernetes.io/projected/8695216c-8df4-4b31-8bd4-11c2567e759f-kube-api-access-s4jfn\") pod \"payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx\" (UID: \"8695216c-8df4-4b31-8bd4-11c2567e759f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:04:19.123875 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.123810    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8695216c-8df4-4b31-8bd4-11c2567e759f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx\" (UID: \"8695216c-8df4-4b31-8bd4-11c2567e759f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:04:19.124412 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.124389    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fc24d3-92d2-4629-8652-776f05907e9c-plugins-config-volume\") pod \"payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck\" (UID: \"c0fc24d3-92d2-4629-8652-776f05907e9c\") " pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:04:19.124576 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.124554    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8695216c-8df4-4b31-8bd4-11c2567e759f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx\" (UID: \"8695216c-8df4-4b31-8bd4-11c2567e759f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:04:19.131249 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.131220    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4jfn\" (UniqueName: \"kubernetes.io/projected/8695216c-8df4-4b31-8bd4-11c2567e759f-kube-api-access-s4jfn\") pod \"payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx\" (UID: \"8695216c-8df4-4b31-8bd4-11c2567e759f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:04:19.131378 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.131357    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxkmm\" (UniqueName: \"kubernetes.io/projected/c0fc24d3-92d2-4629-8652-776f05907e9c-kube-api-access-vxkmm\") pod \"payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck\" (UID: \"c0fc24d3-92d2-4629-8652-776f05907e9c\") " pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:04:19.224805 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.224777    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:04:19.243125 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.243094    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:04:19.275985 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.275957    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:04:19.529187 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.529161    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"]
Sep 10 00:04:19.530765 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:19.530741    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8695216c_8df4_4b31_8bd4_11c2567e759f.slice/crio-4bc0f0e95b784b3fd10807c94c2d3cd55eb3e92bddf58ef39e2596b642985fa3 WatchSource:0}: Error finding container 4bc0f0e95b784b3fd10807c94c2d3cd55eb3e92bddf58ef39e2596b642985fa3: Status 404 returned error can't find the container with id 4bc0f0e95b784b3fd10807c94c2d3cd55eb3e92bddf58ef39e2596b642985fa3
Sep 10 00:04:19.571724 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:19.571696    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"]
Sep 10 00:04:19.585197 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:19.585173    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fc24d3_92d2_4629_8652_776f05907e9c.slice/crio-16caed652e865ff6f0b37dbda05ec598928df7fe025f3bb1584204f1d49454f8 WatchSource:0}: Error finding container 16caed652e865ff6f0b37dbda05ec598928df7fe025f3bb1584204f1d49454f8: Status 404 returned error can't find the container with id 16caed652e865ff6f0b37dbda05ec598928df7fe025f3bb1584204f1d49454f8
Sep 10 00:04:20.233158 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:20.233124    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:04:20.233524 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:20.233202    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:04:20.290281 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:20.289950    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck" event={"ID":"c0fc24d3-92d2-4629-8652-776f05907e9c","Type":"ContainerStarted","Data":"ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d"}
Sep 10 00:04:20.290281 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:20.290017    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck" event={"ID":"c0fc24d3-92d2-4629-8652-776f05907e9c","Type":"ContainerStarted","Data":"16caed652e865ff6f0b37dbda05ec598928df7fe025f3bb1584204f1d49454f8"}
Sep 10 00:04:20.290281 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:20.290075    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:04:20.292566 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:20.292538    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx" event={"ID":"8695216c-8df4-4b31-8bd4-11c2567e759f","Type":"ContainerStarted","Data":"7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1"}
Sep 10 00:04:20.292718 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:20.292591    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:04:20.292718 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:20.292614    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx" event={"ID":"8695216c-8df4-4b31-8bd4-11c2567e759f","Type":"ContainerStarted","Data":"4bc0f0e95b784b3fd10807c94c2d3cd55eb3e92bddf58ef39e2596b642985fa3"}
Sep 10 00:04:20.305430 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:20.305383    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck" podStartSLOduration=2.305368612 podStartE2EDuration="2.305368612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:20.304599159 +0000 UTC m=+940.555735203" watchObservedRunningTime="2026-09-10 00:04:20.305368612 +0000 UTC m=+940.556504653"
Sep 10 00:04:20.322715 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:20.322665    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx" podStartSLOduration=2.322649462 podStartE2EDuration="2.322649462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:20.321610982 +0000 UTC m=+940.572747029" watchObservedRunningTime="2026-09-10 00:04:20.322649462 +0000 UTC m=+940.573785504"
Sep 10 00:04:21.238381 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:21.238347    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:04:22.243792 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:22.243757    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:04:26.268725 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:26.268693    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:31.300679 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:31.300646    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:04:31.301088 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:31.300717    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:04:36.807115 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.807078    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"]
Sep 10 00:04:36.812137 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.812118    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:04:36.815532 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.815513    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a6d968\""
Sep 10 00:04:36.815639 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.815564    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a6d968-dockercfg-prf7r\""
Sep 10 00:04:36.825265 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.825244    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"]
Sep 10 00:04:36.852239 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.852217    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"]
Sep 10 00:04:36.855690 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.855673    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:04:36.869290 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.869272    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"]
Sep 10 00:04:36.897599 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.897579    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c2e4e5-8e38-4a87-b681-3aede504f838-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56\" (UID: \"27c2e4e5-8e38-4a87-b681-3aede504f838\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:04:36.897695 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.897640    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7psv\" (UniqueName: \"kubernetes.io/projected/27c2e4e5-8e38-4a87-b681-3aede504f838-kube-api-access-l7psv\") pod \"payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56\" (UID: \"27c2e4e5-8e38-4a87-b681-3aede504f838\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:04:36.999168 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.999136    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp\" (UID: \"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:04:36.999315 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.999188    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c2e4e5-8e38-4a87-b681-3aede504f838-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56\" (UID: \"27c2e4e5-8e38-4a87-b681-3aede504f838\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:04:36.999315 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.999276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7psv\" (UniqueName: \"kubernetes.io/projected/27c2e4e5-8e38-4a87-b681-3aede504f838-kube-api-access-l7psv\") pod \"payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56\" (UID: \"27c2e4e5-8e38-4a87-b681-3aede504f838\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:04:36.999388 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.999327    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd2h6\" (UniqueName: \"kubernetes.io/projected/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-kube-api-access-zd2h6\") pod \"payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp\" (UID: \"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:04:36.999964 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:36.999946    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c2e4e5-8e38-4a87-b681-3aede504f838-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56\" (UID: \"27c2e4e5-8e38-4a87-b681-3aede504f838\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:04:37.007436 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:37.007408    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7psv\" (UniqueName: \"kubernetes.io/projected/27c2e4e5-8e38-4a87-b681-3aede504f838-kube-api-access-l7psv\") pod \"payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56\" (UID: \"27c2e4e5-8e38-4a87-b681-3aede504f838\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:04:37.100526 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:37.100466    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp\" (UID: \"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:04:37.100659 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:37.100528    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd2h6\" (UniqueName: \"kubernetes.io/projected/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-kube-api-access-zd2h6\") pod \"payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp\" (UID: \"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:04:37.101155 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:37.101134    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp\" (UID: \"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:04:37.108254 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:37.108232    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd2h6\" (UniqueName: \"kubernetes.io/projected/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-kube-api-access-zd2h6\") pod \"payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp\" (UID: \"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:04:37.123634 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:37.123611    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:04:37.165577 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:37.165540    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:04:37.400319 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:37.400285    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"]
Sep 10 00:04:37.401677 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:37.401648    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c2e4e5_8e38_4a87_b681_3aede504f838.slice/crio-1f831c1de71497a1c7cb11ce2f1e521876981b7698a26cfa1f0bbf481e2f77f0 WatchSource:0}: Error finding container 1f831c1de71497a1c7cb11ce2f1e521876981b7698a26cfa1f0bbf481e2f77f0: Status 404 returned error can't find the container with id 1f831c1de71497a1c7cb11ce2f1e521876981b7698a26cfa1f0bbf481e2f77f0
Sep 10 00:04:37.445171 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:37.445143    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"]
Sep 10 00:04:37.446964 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:37.446935    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9ad54e_34e0_4f57_9a5b_dbecd3de6e43.slice/crio-242f1bd8d8f4bb164203697bfbadc4444aa99224d13b3fcf2a7f794b94d4920f WatchSource:0}: Error finding container 242f1bd8d8f4bb164203697bfbadc4444aa99224d13b3fcf2a7f794b94d4920f: Status 404 returned error can't find the container with id 242f1bd8d8f4bb164203697bfbadc4444aa99224d13b3fcf2a7f794b94d4920f
Sep 10 00:04:38.390300 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:38.390265    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56" event={"ID":"27c2e4e5-8e38-4a87-b681-3aede504f838","Type":"ContainerStarted","Data":"031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5"}
Sep 10 00:04:38.390783 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:38.390323    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56" event={"ID":"27c2e4e5-8e38-4a87-b681-3aede504f838","Type":"ContainerStarted","Data":"1f831c1de71497a1c7cb11ce2f1e521876981b7698a26cfa1f0bbf481e2f77f0"}
Sep 10 00:04:38.390783 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:38.390388    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:04:38.391980 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:38.391952    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp" event={"ID":"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43","Type":"ContainerStarted","Data":"167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b"}
Sep 10 00:04:38.392109 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:38.392004    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp" event={"ID":"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43","Type":"ContainerStarted","Data":"242f1bd8d8f4bb164203697bfbadc4444aa99224d13b3fcf2a7f794b94d4920f"}
Sep 10 00:04:38.392109 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:38.392085    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:04:38.407932 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:38.407889    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56" podStartSLOduration=2.407872094 podStartE2EDuration="2.407872094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:38.405295409 +0000 UTC m=+958.656431452" watchObservedRunningTime="2026-09-10 00:04:38.407872094 +0000 UTC m=+958.659008139"
Sep 10 00:04:38.424268 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:38.424228    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp" podStartSLOduration=2.424216534 podStartE2EDuration="2.424216534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:38.421831823 +0000 UTC m=+958.672967868" watchObservedRunningTime="2026-09-10 00:04:38.424216534 +0000 UTC m=+958.675352576"
Sep 10 00:04:42.057544 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.057509    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"]
Sep 10 00:04:42.062915 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.062891    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.066600 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.066576    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"simulator-w0-01b7d5-kserve-self-signed-certs\""
Sep 10 00:04:42.066740 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.066577    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"kube-root-ca.crt\""
Sep 10 00:04:42.066740 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.066620    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"openshift-service-ca.crt\""
Sep 10 00:04:42.066740 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.066577    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"default-dockercfg-hkwkb\""
Sep 10 00:04:42.070240 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.070220    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"]
Sep 10 00:04:42.145533 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.145498    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-home\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.145533 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.145541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tls-certs\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.145733 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.145561    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tmp-dir\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.145733 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.145699    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-dshm\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.145850 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.145749    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-model-cache\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.145850 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.145785    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjlk\" (UniqueName: \"kubernetes.io/projected/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-kube-api-access-8mjlk\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.246180 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.246153    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tmp-dir\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.246297 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.246224    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-dshm\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.246297 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.246262    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-model-cache\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.246297 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.246282    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjlk\" (UniqueName: \"kubernetes.io/projected/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-kube-api-access-8mjlk\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.246474 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.246314    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-home\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.246474 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.246347    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tls-certs\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.246589 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.246495    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tmp-dir\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.246640 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.246626    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-model-cache\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.246680 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.246663    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-home\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.248567 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.248541    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-dshm\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.248826 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.248809    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tls-certs\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.253477 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.253449    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjlk\" (UniqueName: \"kubernetes.io/projected/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-kube-api-access-8mjlk\") pod \"simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") " pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.372919 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.372825    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:42.644096 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:42.644072    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"]
Sep 10 00:04:42.645475 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:42.645448    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf09d0d1_b106_49ad_ad92_6724c0d67aa1.slice/crio-a47f869b3a6c29f59647c360643a320c6ce5f0838b60359ccb2719ed78a13063 WatchSource:0}: Error finding container a47f869b3a6c29f59647c360643a320c6ce5f0838b60359ccb2719ed78a13063: Status 404 returned error can't find the container with id a47f869b3a6c29f59647c360643a320c6ce5f0838b60359ccb2719ed78a13063
Sep 10 00:04:43.422187 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:43.422151    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k" event={"ID":"bf09d0d1-b106-49ad-ad92-6724c0d67aa1","Type":"ContainerStarted","Data":"3a5c7939aa34794cc24104ce1b5157fbad9ad6c1da713d9abe16c71afa6609d0"}
Sep 10 00:04:43.422688 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:43.422199    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k" event={"ID":"bf09d0d1-b106-49ad-ad92-6724c0d67aa1","Type":"ContainerStarted","Data":"a47f869b3a6c29f59647c360643a320c6ce5f0838b60359ccb2719ed78a13063"}
Sep 10 00:04:43.439564 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:43.439516    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k" podStartSLOduration=1.439501193 podStartE2EDuration="1.439501193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:04:43.437527215 +0000 UTC m=+963.688663259" watchObservedRunningTime="2026-09-10 00:04:43.439501193 +0000 UTC m=+963.690637267"
Sep 10 00:04:49.400425 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:49.400391    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:04:49.400816 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:49.400626    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:04:52.373226 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:52.373183    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:52.373226 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:52.373241    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:52.386624 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:52.386592    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:52.483369 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:52.483348    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:04:54.767374 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:54.767339    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"]
Sep 10 00:04:54.767846 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:54.767583    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w" podUID="b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b" containerName="payload-pre-processing" containerID="cri-o://09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e" gracePeriod=30
Sep 10 00:04:55.039834 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.039760    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:55.165433 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.165401    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-plugins-config-volume\") pod \"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b\" (UID: \"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b\") "
Sep 10 00:04:55.165598 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.165495    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwfbq\" (UniqueName: \"kubernetes.io/projected/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-kube-api-access-jwfbq\") pod \"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b\" (UID: \"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b\") "
Sep 10 00:04:55.165767 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.165743    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b" (UID: "b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:04:55.167540 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.167520    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-kube-api-access-jwfbq" (OuterVolumeSpecName: "kube-api-access-jwfbq") pod "b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b" (UID: "b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b"). InnerVolumeSpecName "kube-api-access-jwfbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:04:55.266667 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.266641    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:04:55.266667 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.266662    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwfbq\" (UniqueName: \"kubernetes.io/projected/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b-kube-api-access-jwfbq\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:04:55.488889 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.488858    2556 generic.go:356] "Generic (PLEG): container finished" podID="b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b" containerID="09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e" exitCode=0
Sep 10 00:04:55.489026 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.488906    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"
Sep 10 00:04:55.489026 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.488941    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w" event={"ID":"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b","Type":"ContainerDied","Data":"09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e"}
Sep 10 00:04:55.489026 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.489004    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w" event={"ID":"b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b","Type":"ContainerDied","Data":"7590e8cf00ee369b2407bdd6408da24ce256a10955768e04c5b3a424b84ed096"}
Sep 10 00:04:55.489190 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.489028    2556 scope.go:117] "RemoveContainer" containerID="09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e"
Sep 10 00:04:55.499599 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.499583    2556 scope.go:117] "RemoveContainer" containerID="09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e"
Sep 10 00:04:55.499856 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:04:55.499841    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e\": container with ID starting with 09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e not found: ID does not exist" containerID="09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e"
Sep 10 00:04:55.499922 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.499863    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e"} err="failed to get container status \"09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e\": rpc error: code = NotFound desc = could not find container \"09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e\": container with ID starting with 09da62c8b52feb1c8114ac70b42336bddaccdf08aad5e0bb3caddeaf2fe5d88e not found: ID does not exist"
Sep 10 00:04:55.519485 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.519430    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"]
Sep 10 00:04:55.521531 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:55.521514    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46484ed0-6cc6f5fddb-b547w"]
Sep 10 00:04:56.268274 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:56.268240    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b" path="/var/lib/kubelet/pods/b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b/volumes"
Sep 10 00:04:59.053926 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.053892    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"]
Sep 10 00:04:59.054572 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.054549    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b" containerName="payload-pre-processing"
Sep 10 00:04:59.054669 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.054574    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b" containerName="payload-pre-processing"
Sep 10 00:04:59.054734 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.054720    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6ddbbb2-69ed-4dc2-9f0f-98ec8e07be0b" containerName="payload-pre-processing"
Sep 10 00:04:59.059361 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.059341    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:04:59.061878 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.061858    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e214cef3-dockercfg-vjljx\""
Sep 10 00:04:59.061985 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.061953    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e214cef3\""
Sep 10 00:04:59.068604 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.068575    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"]
Sep 10 00:04:59.098125 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.098103    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb\" (UID: \"8a2ef6c1-f89a-40a5-aac0-110b50c3c442\") " pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:04:59.098214 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.098185    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m458\" (UniqueName: \"kubernetes.io/projected/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-kube-api-access-2m458\") pod \"payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb\" (UID: \"8a2ef6c1-f89a-40a5-aac0-110b50c3c442\") " pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:04:59.198626 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.198594    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb\" (UID: \"8a2ef6c1-f89a-40a5-aac0-110b50c3c442\") " pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:04:59.198758 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.198657    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m458\" (UniqueName: \"kubernetes.io/projected/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-kube-api-access-2m458\") pod \"payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb\" (UID: \"8a2ef6c1-f89a-40a5-aac0-110b50c3c442\") " pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:04:59.199300 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.199280    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb\" (UID: \"8a2ef6c1-f89a-40a5-aac0-110b50c3c442\") " pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:04:59.205756 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.205729    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m458\" (UniqueName: \"kubernetes.io/projected/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-kube-api-access-2m458\") pod \"payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb\" (UID: \"8a2ef6c1-f89a-40a5-aac0-110b50c3c442\") " pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:04:59.370878 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.370807    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:04:59.644587 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:04:59.644564    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"]
Sep 10 00:04:59.646329 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:04:59.646299    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2ef6c1_f89a_40a5_aac0_110b50c3c442.slice/crio-fa7aa2d2bb096b53171beff8f734c9e44c553f23c40f357643a6e236fecaa0e7 WatchSource:0}: Error finding container fa7aa2d2bb096b53171beff8f734c9e44c553f23c40f357643a6e236fecaa0e7: Status 404 returned error can't find the container with id fa7aa2d2bb096b53171beff8f734c9e44c553f23c40f357643a6e236fecaa0e7
Sep 10 00:05:00.519371 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:00.519337    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb" event={"ID":"8a2ef6c1-f89a-40a5-aac0-110b50c3c442","Type":"ContainerStarted","Data":"aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d"}
Sep 10 00:05:00.519755 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:00.519431    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb" event={"ID":"8a2ef6c1-f89a-40a5-aac0-110b50c3c442","Type":"ContainerStarted","Data":"fa7aa2d2bb096b53171beff8f734c9e44c553f23c40f357643a6e236fecaa0e7"}
Sep 10 00:05:00.519755 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:00.519457    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:05:00.535180 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:00.535105    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb" podStartSLOduration=1.535084435 podStartE2EDuration="1.535084435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:05:00.534713525 +0000 UTC m=+980.785849569" watchObservedRunningTime="2026-09-10 00:05:00.535084435 +0000 UTC m=+980.786220470"
Sep 10 00:05:11.528049 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:11.527976    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:05:29.019529 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.019490    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"]
Sep 10 00:05:29.019963 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.019732    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb" podUID="8a2ef6c1-f89a-40a5-aac0-110b50c3c442" containerName="payload-processing" containerID="cri-o://aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d" gracePeriod=30
Sep 10 00:05:29.269793 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.269738    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:05:29.346717 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.346690    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m458\" (UniqueName: \"kubernetes.io/projected/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-kube-api-access-2m458\") pod \"8a2ef6c1-f89a-40a5-aac0-110b50c3c442\" (UID: \"8a2ef6c1-f89a-40a5-aac0-110b50c3c442\") "
Sep 10 00:05:29.346871 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.346727    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-plugins-config-volume\") pod \"8a2ef6c1-f89a-40a5-aac0-110b50c3c442\" (UID: \"8a2ef6c1-f89a-40a5-aac0-110b50c3c442\") "
Sep 10 00:05:29.347149 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.347115    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a2ef6c1-f89a-40a5-aac0-110b50c3c442" (UID: "8a2ef6c1-f89a-40a5-aac0-110b50c3c442"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:05:29.348756 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.348722    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-kube-api-access-2m458" (OuterVolumeSpecName: "kube-api-access-2m458") pod "8a2ef6c1-f89a-40a5-aac0-110b50c3c442" (UID: "8a2ef6c1-f89a-40a5-aac0-110b50c3c442"). InnerVolumeSpecName "kube-api-access-2m458". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:05:29.448009 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.447981    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m458\" (UniqueName: \"kubernetes.io/projected/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-kube-api-access-2m458\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:05:29.448009 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.448004    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2ef6c1-f89a-40a5-aac0-110b50c3c442-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:05:29.680534 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.680462    2556 generic.go:356] "Generic (PLEG): container finished" podID="8a2ef6c1-f89a-40a5-aac0-110b50c3c442" containerID="aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d" exitCode=0
Sep 10 00:05:29.680534 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.680501    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"
Sep 10 00:05:29.680534 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.680517    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb" event={"ID":"8a2ef6c1-f89a-40a5-aac0-110b50c3c442","Type":"ContainerDied","Data":"aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d"}
Sep 10 00:05:29.680753 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.680582    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb" event={"ID":"8a2ef6c1-f89a-40a5-aac0-110b50c3c442","Type":"ContainerDied","Data":"fa7aa2d2bb096b53171beff8f734c9e44c553f23c40f357643a6e236fecaa0e7"}
Sep 10 00:05:29.680753 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.680603    2556 scope.go:117] "RemoveContainer" containerID="aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d"
Sep 10 00:05:29.689649 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.689633    2556 scope.go:117] "RemoveContainer" containerID="aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d"
Sep 10 00:05:29.689906 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:05:29.689887    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d\": container with ID starting with aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d not found: ID does not exist" containerID="aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d"
Sep 10 00:05:29.689957 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.689916    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d"} err="failed to get container status \"aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d\": rpc error: code = NotFound desc = could not find container \"aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d\": container with ID starting with aaf916628a42eac490aa0b00fac027772a068c748b2a03569e18cc0a93acee0d not found: ID does not exist"
Sep 10 00:05:29.701777 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.701755    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"]
Sep 10 00:05:29.707953 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:29.707927    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e214cef3-7b4dfb598-hctcb"]
Sep 10 00:05:30.268185 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:30.268155    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a2ef6c1-f89a-40a5-aac0-110b50c3c442" path="/var/lib/kubelet/pods/8a2ef6c1-f89a-40a5-aac0-110b50c3c442/volumes"
Sep 10 00:05:34.086309 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.086278    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"]
Sep 10 00:05:34.086729 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.086716    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a2ef6c1-f89a-40a5-aac0-110b50c3c442" containerName="payload-processing"
Sep 10 00:05:34.086774 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.086730    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a2ef6c1-f89a-40a5-aac0-110b50c3c442" containerName="payload-processing"
Sep 10 00:05:34.086819 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.086810    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a2ef6c1-f89a-40a5-aac0-110b50c3c442" containerName="payload-processing"
Sep 10 00:05:34.091429 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.091413    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:05:34.093718 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.093693    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a2f357ac-dockercfg-nfd4n\""
Sep 10 00:05:34.093839 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.093726    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a2f357ac\""
Sep 10 00:05:34.102067 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.102028    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"]
Sep 10 00:05:34.187265 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.187230    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2\" (UID: \"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791\") " pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:05:34.187420 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.187294    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbzxx\" (UniqueName: \"kubernetes.io/projected/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-kube-api-access-dbzxx\") pod \"payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2\" (UID: \"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791\") " pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:05:34.288442 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.288405    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2\" (UID: \"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791\") " pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:05:34.288609 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.288470    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbzxx\" (UniqueName: \"kubernetes.io/projected/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-kube-api-access-dbzxx\") pod \"payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2\" (UID: \"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791\") " pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:05:34.289322 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.289296    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-plugins-config-volume\") pod \"payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2\" (UID: \"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791\") " pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:05:34.295835 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.295810    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbzxx\" (UniqueName: \"kubernetes.io/projected/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-kube-api-access-dbzxx\") pod \"payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2\" (UID: \"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791\") " pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:05:34.408651 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.408591    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:05:34.678400 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.678376    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"]
Sep 10 00:05:34.680129 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:05:34.680102    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd8b987_ad6c_4fd5_8b64_b21a8ce4a791.slice/crio-7288979ef8d16856d7d2e7800935025123d173e117f37f0055fb1ee19ba37e3b WatchSource:0}: Error finding container 7288979ef8d16856d7d2e7800935025123d173e117f37f0055fb1ee19ba37e3b: Status 404 returned error can't find the container with id 7288979ef8d16856d7d2e7800935025123d173e117f37f0055fb1ee19ba37e3b
Sep 10 00:05:34.710046 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:34.710022    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2" event={"ID":"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791","Type":"ContainerStarted","Data":"7288979ef8d16856d7d2e7800935025123d173e117f37f0055fb1ee19ba37e3b"}
Sep 10 00:05:35.718790 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:35.718747    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2" event={"ID":"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791","Type":"ContainerStarted","Data":"dd009bfd537b8c897eb98e90dba31b4b2f391181e1e43dd51354e5dd902b0556"}
Sep 10 00:05:35.719223 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:35.718814    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:05:35.736218 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:35.736172    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2" podStartSLOduration=1.736157156 podStartE2EDuration="1.736157156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:05:35.734360408 +0000 UTC m=+1015.985496452" watchObservedRunningTime="2026-09-10 00:05:35.736157156 +0000 UTC m=+1015.987293198"
Sep 10 00:05:46.726722 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:46.726693    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:05:52.053500 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.053461    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"]
Sep 10 00:05:52.058755 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.058735    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.062379 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.062354    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f0f965\"/\"default-dockercfg-fhgcc\""
Sep 10 00:05:52.062599 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.062368    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f0f965\"/\"kube-root-ca.crt\""
Sep 10 00:05:52.062711 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.062385    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f0f965\"/\"test-model-f0f965-kserve-self-signed-certs\""
Sep 10 00:05:52.062775 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.062442    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f0f965\"/\"openshift-service-ca.crt\""
Sep 10 00:05:52.064982 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.064961    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"]
Sep 10 00:05:52.242859 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.242829    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-model-cache\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.243039 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.242870    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-tmp-dir\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.243039 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.242934    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-home\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.243202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.243042    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qxsz\" (UniqueName: \"kubernetes.io/projected/6a424378-3af5-434e-a810-5621681078c4-kube-api-access-5qxsz\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.243202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.243134    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-dshm\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.243202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.243162    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a424378-3af5-434e-a810-5621681078c4-tls-certs\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.344201 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.344138    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a424378-3af5-434e-a810-5621681078c4-tls-certs\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.344201 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.344191    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-model-cache\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.344352 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.344225    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-tmp-dir\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.344352 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.344259    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-home\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.344352 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.344319    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qxsz\" (UniqueName: \"kubernetes.io/projected/6a424378-3af5-434e-a810-5621681078c4-kube-api-access-5qxsz\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.344474 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.344369    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-dshm\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.344690 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.344669    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-model-cache\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.344756 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.344703    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-tmp-dir\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.344756 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.344715    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-home\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.346542 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.346518    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-dshm\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.346675 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.346659    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a424378-3af5-434e-a810-5621681078c4-tls-certs\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.351782 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.351765    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qxsz\" (UniqueName: \"kubernetes.io/projected/6a424378-3af5-434e-a810-5621681078c4-kube-api-access-5qxsz\") pod \"test-model-f0f965-kserve-784f76ff69-7xrsh\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") " pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.370193 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.370169    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:05:52.639658 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.639637    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"]
Sep 10 00:05:52.641537 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:05:52.641499    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a424378_3af5_434e_a810_5621681078c4.slice/crio-6195a910888b575822fd78e62043e4803b53ab843534966736ee7919e544de59 WatchSource:0}: Error finding container 6195a910888b575822fd78e62043e4803b53ab843534966736ee7919e544de59: Status 404 returned error can't find the container with id 6195a910888b575822fd78e62043e4803b53ab843534966736ee7919e544de59
Sep 10 00:05:52.811688 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.811650    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh" event={"ID":"6a424378-3af5-434e-a810-5621681078c4","Type":"ContainerStarted","Data":"9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d"}
Sep 10 00:05:52.811846 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.811701    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh" event={"ID":"6a424378-3af5-434e-a810-5621681078c4","Type":"ContainerStarted","Data":"6195a910888b575822fd78e62043e4803b53ab843534966736ee7919e544de59"}
Sep 10 00:05:52.827739 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:05:52.827687    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh" podStartSLOduration=0.827669047 podStartE2EDuration="827.669047ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:05:52.825385057 +0000 UTC m=+1033.076521099" watchObservedRunningTime="2026-09-10 00:05:52.827669047 +0000 UTC m=+1033.078805092"
Sep 10 00:06:02.371313 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:02.371210    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:06:02.371313 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:02.371280    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:06:02.387989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:02.387957    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:06:02.874723 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:02.874694    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:06:09.527265 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:09.527235    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"]
Sep 10 00:06:09.530802 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:09.530784    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"
Sep 10 00:06:09.533043 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:09.533023    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:06:09.533173 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:09.533029    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:06:09.537618 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:09.537594    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"]
Sep 10 00:06:09.586446 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:09.586412    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e3c289-490f-469b-8633-cc92e57126c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-39c7cb-w6772\" (UID: \"88e3c289-490f-469b-8633-cc92e57126c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"
Sep 10 00:06:09.687489 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:09.687450    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e3c289-490f-469b-8633-cc92e57126c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-39c7cb-w6772\" (UID: \"88e3c289-490f-469b-8633-cc92e57126c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"
Sep 10 00:06:09.688100 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:09.688051    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e3c289-490f-469b-8633-cc92e57126c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-39c7cb-w6772\" (UID: \"88e3c289-490f-469b-8633-cc92e57126c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"
Sep 10 00:06:09.845627 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:09.845545    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"
Sep 10 00:06:10.114279 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:10.114254    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"]
Sep 10 00:06:10.116070 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:06:10.116023    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e3c289_490f_469b_8633_cc92e57126c8.slice/crio-5a8ed1c4fbc7b5ed3cc6361f96cde985897742099bab91a96d2352153ce248bf WatchSource:0}: Error finding container 5a8ed1c4fbc7b5ed3cc6361f96cde985897742099bab91a96d2352153ce248bf: Status 404 returned error can't find the container with id 5a8ed1c4fbc7b5ed3cc6361f96cde985897742099bab91a96d2352153ce248bf
Sep 10 00:06:10.914922 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:10.914877    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772" event={"ID":"88e3c289-490f-469b-8633-cc92e57126c8","Type":"ContainerStarted","Data":"5a8ed1c4fbc7b5ed3cc6361f96cde985897742099bab91a96d2352153ce248bf"}
Sep 10 00:06:12.886633 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:06:12.886598    2556 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e3c289_490f_469b_8633_cc92e57126c8.slice/crio-conmon-4b5bdb629a3fddf01269bc3e106b0de0835c9439dafc0eba612b3a44d9268ddb.scope\": RecentStats: unable to find data in memory cache]"
Sep 10 00:06:12.927742 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:12.927706    2556 generic.go:356] "Generic (PLEG): container finished" podID="88e3c289-490f-469b-8633-cc92e57126c8" containerID="4b5bdb629a3fddf01269bc3e106b0de0835c9439dafc0eba612b3a44d9268ddb" exitCode=0
Sep 10 00:06:12.927881 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:12.927748    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772" event={"ID":"88e3c289-490f-469b-8633-cc92e57126c8","Type":"ContainerDied","Data":"4b5bdb629a3fddf01269bc3e106b0de0835c9439dafc0eba612b3a44d9268ddb"}
Sep 10 00:06:14.067189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:14.067165    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"
Sep 10 00:06:14.123816 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:14.123788    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e3c289-490f-469b-8633-cc92e57126c8-maas-api-service-ca\") pod \"88e3c289-490f-469b-8633-cc92e57126c8\" (UID: \"88e3c289-490f-469b-8633-cc92e57126c8\") "
Sep 10 00:06:14.124143 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:14.124121    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e3c289-490f-469b-8633-cc92e57126c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88e3c289-490f-469b-8633-cc92e57126c8" (UID: "88e3c289-490f-469b-8633-cc92e57126c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:06:14.224784 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:14.224756    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e3c289-490f-469b-8633-cc92e57126c8-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:06:14.939735 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:14.939708    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"
Sep 10 00:06:14.939735 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:14.939740    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772" event={"ID":"88e3c289-490f-469b-8633-cc92e57126c8","Type":"ContainerDied","Data":"5a8ed1c4fbc7b5ed3cc6361f96cde985897742099bab91a96d2352153ce248bf"}
Sep 10 00:06:14.939993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:14.939785    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a8ed1c4fbc7b5ed3cc6361f96cde985897742099bab91a96d2352153ce248bf"
Sep 10 00:06:18.844754 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:18.844713    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"]
Sep 10 00:06:18.845629 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:18.845592    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2" podUID="9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791" containerName="payload-processing" containerID="cri-o://dd009bfd537b8c897eb98e90dba31b4b2f391181e1e43dd51354e5dd902b0556" gracePeriod=30
Sep 10 00:06:18.962914 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:18.962877    2556 generic.go:356] "Generic (PLEG): container finished" podID="9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791" containerID="dd009bfd537b8c897eb98e90dba31b4b2f391181e1e43dd51354e5dd902b0556" exitCode=0
Sep 10 00:06:18.963048 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:18.962950    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2" event={"ID":"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791","Type":"ContainerDied","Data":"dd009bfd537b8c897eb98e90dba31b4b2f391181e1e43dd51354e5dd902b0556"}
Sep 10 00:06:19.114330 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.114301    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:06:19.170398 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.170372    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbzxx\" (UniqueName: \"kubernetes.io/projected/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-kube-api-access-dbzxx\") pod \"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791\" (UID: \"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791\") "
Sep 10 00:06:19.170549 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.170433    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-plugins-config-volume\") pod \"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791\" (UID: \"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791\") "
Sep 10 00:06:19.170832 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.170808    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791" (UID: "9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:06:19.172551 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.172522    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-kube-api-access-dbzxx" (OuterVolumeSpecName: "kube-api-access-dbzxx") pod "9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791" (UID: "9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791"). InnerVolumeSpecName "kube-api-access-dbzxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:06:19.272211 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.272170    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbzxx\" (UniqueName: \"kubernetes.io/projected/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-kube-api-access-dbzxx\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:06:19.272211 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.272207    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:06:19.970523 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.970490    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"
Sep 10 00:06:19.970933 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.970489    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2" event={"ID":"9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791","Type":"ContainerDied","Data":"7288979ef8d16856d7d2e7800935025123d173e117f37f0055fb1ee19ba37e3b"}
Sep 10 00:06:19.970933 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.970627    2556 scope.go:117] "RemoveContainer" containerID="dd009bfd537b8c897eb98e90dba31b4b2f391181e1e43dd51354e5dd902b0556"
Sep 10 00:06:19.992812 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.992782    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"]
Sep 10 00:06:19.996272 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:19.996251    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a2f357ac-85648769bc-lr4p2"]
Sep 10 00:06:20.272154 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.272080    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791" path="/var/lib/kubelet/pods/9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791/volumes"
Sep 10 00:06:20.358519 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.358478    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"]
Sep 10 00:06:20.358776 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.358741    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck" podUID="c0fc24d3-92d2-4629-8652-776f05907e9c" containerName="payload-processing" containerID="cri-o://ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d" gracePeriod=30
Sep 10 00:06:20.363786 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.363664    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"]
Sep 10 00:06:20.364014 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.363980    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx" podUID="8695216c-8df4-4b31-8bd4-11c2567e759f" containerName="payload-pre-processing" containerID="cri-o://7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1" gracePeriod=30
Sep 10 00:06:20.535956 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.535859    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"]
Sep 10 00:06:20.539856 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.539829    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-39c7cb-w6772"]
Sep 10 00:06:20.646513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.646486    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:06:20.649682 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.649663    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:06:20.687661 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.687629    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fc24d3-92d2-4629-8652-776f05907e9c-plugins-config-volume\") pod \"c0fc24d3-92d2-4629-8652-776f05907e9c\" (UID: \"c0fc24d3-92d2-4629-8652-776f05907e9c\") "
Sep 10 00:06:20.687661 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.687661    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4jfn\" (UniqueName: \"kubernetes.io/projected/8695216c-8df4-4b31-8bd4-11c2567e759f-kube-api-access-s4jfn\") pod \"8695216c-8df4-4b31-8bd4-11c2567e759f\" (UID: \"8695216c-8df4-4b31-8bd4-11c2567e759f\") "
Sep 10 00:06:20.687868 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.687683    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxkmm\" (UniqueName: \"kubernetes.io/projected/c0fc24d3-92d2-4629-8652-776f05907e9c-kube-api-access-vxkmm\") pod \"c0fc24d3-92d2-4629-8652-776f05907e9c\" (UID: \"c0fc24d3-92d2-4629-8652-776f05907e9c\") "
Sep 10 00:06:20.687868 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.687745    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8695216c-8df4-4b31-8bd4-11c2567e759f-plugins-config-volume\") pod \"8695216c-8df4-4b31-8bd4-11c2567e759f\" (UID: \"8695216c-8df4-4b31-8bd4-11c2567e759f\") "
Sep 10 00:06:20.688171 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.688144    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fc24d3-92d2-4629-8652-776f05907e9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0fc24d3-92d2-4629-8652-776f05907e9c" (UID: "c0fc24d3-92d2-4629-8652-776f05907e9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:06:20.688299 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.688218    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8695216c-8df4-4b31-8bd4-11c2567e759f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8695216c-8df4-4b31-8bd4-11c2567e759f" (UID: "8695216c-8df4-4b31-8bd4-11c2567e759f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:06:20.689872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.689846    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8695216c-8df4-4b31-8bd4-11c2567e759f-kube-api-access-s4jfn" (OuterVolumeSpecName: "kube-api-access-s4jfn") pod "8695216c-8df4-4b31-8bd4-11c2567e759f" (UID: "8695216c-8df4-4b31-8bd4-11c2567e759f"). InnerVolumeSpecName "kube-api-access-s4jfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:06:20.689949 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.689857    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fc24d3-92d2-4629-8652-776f05907e9c-kube-api-access-vxkmm" (OuterVolumeSpecName: "kube-api-access-vxkmm") pod "c0fc24d3-92d2-4629-8652-776f05907e9c" (UID: "c0fc24d3-92d2-4629-8652-776f05907e9c"). InnerVolumeSpecName "kube-api-access-vxkmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:06:20.789341 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.789243    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0fc24d3-92d2-4629-8652-776f05907e9c-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:06:20.789341 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.789279    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4jfn\" (UniqueName: \"kubernetes.io/projected/8695216c-8df4-4b31-8bd4-11c2567e759f-kube-api-access-s4jfn\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:06:20.789341 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.789293    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxkmm\" (UniqueName: \"kubernetes.io/projected/c0fc24d3-92d2-4629-8652-776f05907e9c-kube-api-access-vxkmm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:06:20.789341 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.789306    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8695216c-8df4-4b31-8bd4-11c2567e759f-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:06:20.977912 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.977876    2556 generic.go:356] "Generic (PLEG): container finished" podID="c0fc24d3-92d2-4629-8652-776f05907e9c" containerID="ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d" exitCode=0
Sep 10 00:06:20.978359 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.977928    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"
Sep 10 00:06:20.978359 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.977959    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck" event={"ID":"c0fc24d3-92d2-4629-8652-776f05907e9c","Type":"ContainerDied","Data":"ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d"}
Sep 10 00:06:20.978359 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.978017    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck" event={"ID":"c0fc24d3-92d2-4629-8652-776f05907e9c","Type":"ContainerDied","Data":"16caed652e865ff6f0b37dbda05ec598928df7fe025f3bb1584204f1d49454f8"}
Sep 10 00:06:20.978359 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.978045    2556 scope.go:117] "RemoveContainer" containerID="ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d"
Sep 10 00:06:20.987478 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.987185    2556 generic.go:356] "Generic (PLEG): container finished" podID="8695216c-8df4-4b31-8bd4-11c2567e759f" containerID="7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1" exitCode=0
Sep 10 00:06:20.987478 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.987246    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"
Sep 10 00:06:20.987478 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.987301    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx" event={"ID":"8695216c-8df4-4b31-8bd4-11c2567e759f","Type":"ContainerDied","Data":"7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1"}
Sep 10 00:06:20.987478 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.987367    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx" event={"ID":"8695216c-8df4-4b31-8bd4-11c2567e759f","Type":"ContainerDied","Data":"4bc0f0e95b784b3fd10807c94c2d3cd55eb3e92bddf58ef39e2596b642985fa3"}
Sep 10 00:06:20.997715 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.997695    2556 scope.go:117] "RemoveContainer" containerID="ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d"
Sep 10 00:06:20.997978 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:06:20.997956    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d\": container with ID starting with ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d not found: ID does not exist" containerID="ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d"
Sep 10 00:06:20.998043 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.997988    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d"} err="failed to get container status \"ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d\": rpc error: code = NotFound desc = could not find container \"ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d\": container with ID starting with ea9cb9c8332012cb8580a1187432d59ee9d0d45824eb1f9758b43e6485f7723d not found: ID does not exist"
Sep 10 00:06:20.998043 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:20.998011    2556 scope.go:117] "RemoveContainer" containerID="7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1"
Sep 10 00:06:21.005642 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:21.005598    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"]
Sep 10 00:06:21.008513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:21.008499    2556 scope.go:117] "RemoveContainer" containerID="7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1"
Sep 10 00:06:21.008801 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:06:21.008779    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1\": container with ID starting with 7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1 not found: ID does not exist" containerID="7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1"
Sep 10 00:06:21.008853 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:21.008810    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1"} err="failed to get container status \"7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1\": rpc error: code = NotFound desc = could not find container \"7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1\": container with ID starting with 7e52c12b88eb9a2dfa7d3a9b4937363c67f443cb30bf5d2fc5c5e779141b5cf1 not found: ID does not exist"
Sep 10 00:06:21.008853 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:21.008839    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-39c7cb-9694fc7dc-2b7ck"]
Sep 10 00:06:21.019182 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:21.019156    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"]
Sep 10 00:06:21.024771 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:21.024749    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-39c7cb-659966798-pg9xx"]
Sep 10 00:06:22.268316 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:22.268282    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8695216c-8df4-4b31-8bd4-11c2567e759f" path="/var/lib/kubelet/pods/8695216c-8df4-4b31-8bd4-11c2567e759f/volumes"
Sep 10 00:06:22.268683 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:22.268600    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e3c289-490f-469b-8633-cc92e57126c8" path="/var/lib/kubelet/pods/88e3c289-490f-469b-8633-cc92e57126c8/volumes"
Sep 10 00:06:22.268915 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:22.268902    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0fc24d3-92d2-4629-8652-776f05907e9c" path="/var/lib/kubelet/pods/c0fc24d3-92d2-4629-8652-776f05907e9c/volumes"
Sep 10 00:06:35.491123 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491027    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"]
Sep 10 00:06:35.491514 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491499    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791" containerName="payload-processing"
Sep 10 00:06:35.491555 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491515    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791" containerName="payload-processing"
Sep 10 00:06:35.491555 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491531    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8695216c-8df4-4b31-8bd4-11c2567e759f" containerName="payload-pre-processing"
Sep 10 00:06:35.491555 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491537    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8695216c-8df4-4b31-8bd4-11c2567e759f" containerName="payload-pre-processing"
Sep 10 00:06:35.491555 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491546    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e3c289-490f-469b-8633-cc92e57126c8" containerName="revoke-keys"
Sep 10 00:06:35.491555 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491552    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e3c289-490f-469b-8633-cc92e57126c8" containerName="revoke-keys"
Sep 10 00:06:35.491712 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491564    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fc24d3-92d2-4629-8652-776f05907e9c" containerName="payload-processing"
Sep 10 00:06:35.491712 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491570    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fc24d3-92d2-4629-8652-776f05907e9c" containerName="payload-processing"
Sep 10 00:06:35.491712 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491635    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8695216c-8df4-4b31-8bd4-11c2567e759f" containerName="payload-pre-processing"
Sep 10 00:06:35.491712 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491643    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0fc24d3-92d2-4629-8652-776f05907e9c" containerName="payload-processing"
Sep 10 00:06:35.491712 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491652    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bd8b987-ad6c-4fd5-8b64-b21a8ce4a791" containerName="payload-processing"
Sep 10 00:06:35.491712 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.491660    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e3c289-490f-469b-8633-cc92e57126c8" containerName="revoke-keys"
Sep 10 00:06:35.499543 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.499520    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:06:35.501947 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.501921    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-96c64491-dockercfg-dtbm8\""
Sep 10 00:06:35.502089 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.501960    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-96c64491\""
Sep 10 00:06:35.507234 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.507206    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"]
Sep 10 00:06:35.519637 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.519604    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"]
Sep 10 00:06:35.523870 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.523849    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:06:35.534816 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.534792    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"]
Sep 10 00:06:35.628088 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.628041    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkhxp\" (UniqueName: \"kubernetes.io/projected/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-kube-api-access-nkhxp\") pod \"payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5\" (UID: \"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:06:35.628238 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.628200    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5\" (UID: \"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:06:35.628310 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.628292    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee523b67-1f1e-4a8e-9fce-946d946cc63c-plugins-config-volume\") pod \"payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l\" (UID: \"ee523b67-1f1e-4a8e-9fce-946d946cc63c\") " pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:06:35.628398 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.628369    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdfd\" (UniqueName: \"kubernetes.io/projected/ee523b67-1f1e-4a8e-9fce-946d946cc63c-kube-api-access-7hdfd\") pod \"payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l\" (UID: \"ee523b67-1f1e-4a8e-9fce-946d946cc63c\") " pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:06:35.729557 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.729522    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdfd\" (UniqueName: \"kubernetes.io/projected/ee523b67-1f1e-4a8e-9fce-946d946cc63c-kube-api-access-7hdfd\") pod \"payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l\" (UID: \"ee523b67-1f1e-4a8e-9fce-946d946cc63c\") " pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:06:35.729723 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.729569    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkhxp\" (UniqueName: \"kubernetes.io/projected/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-kube-api-access-nkhxp\") pod \"payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5\" (UID: \"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:06:35.729723 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.729620    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5\" (UID: \"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:06:35.729723 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.729658    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee523b67-1f1e-4a8e-9fce-946d946cc63c-plugins-config-volume\") pod \"payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l\" (UID: \"ee523b67-1f1e-4a8e-9fce-946d946cc63c\") " pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:06:35.730454 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.730430    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee523b67-1f1e-4a8e-9fce-946d946cc63c-plugins-config-volume\") pod \"payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l\" (UID: \"ee523b67-1f1e-4a8e-9fce-946d946cc63c\") " pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:06:35.730544 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.730431    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5\" (UID: \"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:06:35.737267 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.737247    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkhxp\" (UniqueName: \"kubernetes.io/projected/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-kube-api-access-nkhxp\") pod \"payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5\" (UID: \"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:06:35.737489 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.737467    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdfd\" (UniqueName: \"kubernetes.io/projected/ee523b67-1f1e-4a8e-9fce-946d946cc63c-kube-api-access-7hdfd\") pod \"payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l\" (UID: \"ee523b67-1f1e-4a8e-9fce-946d946cc63c\") " pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:06:35.811340 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.811287    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:06:35.835197 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:35.835172    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:06:36.085363 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:36.085337    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"]
Sep 10 00:06:36.087238 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:06:36.087214    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28dbb6f6_2815_40c9_ba1c_10c0ce1bf59a.slice/crio-eac301fd1e8ee28cad18108dfc1c6feb50c2771011a753148b7dbd8e09f62f55 WatchSource:0}: Error finding container eac301fd1e8ee28cad18108dfc1c6feb50c2771011a753148b7dbd8e09f62f55: Status 404 returned error can't find the container with id eac301fd1e8ee28cad18108dfc1c6feb50c2771011a753148b7dbd8e09f62f55
Sep 10 00:06:36.119783 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:36.119752    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"]
Sep 10 00:06:37.079223 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:37.079181    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l" event={"ID":"ee523b67-1f1e-4a8e-9fce-946d946cc63c","Type":"ContainerStarted","Data":"26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec"}
Sep 10 00:06:37.079695 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:37.079249    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l" event={"ID":"ee523b67-1f1e-4a8e-9fce-946d946cc63c","Type":"ContainerStarted","Data":"fc61fa843b391021e15855ebe89fcba452ff36da159d91ecf190fa2a1ab26231"}
Sep 10 00:06:37.079695 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:37.079307    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:06:37.081106 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:37.081083    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5" event={"ID":"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a","Type":"ContainerStarted","Data":"746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b"}
Sep 10 00:06:37.081252 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:37.081128    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5" event={"ID":"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a","Type":"ContainerStarted","Data":"eac301fd1e8ee28cad18108dfc1c6feb50c2771011a753148b7dbd8e09f62f55"}
Sep 10 00:06:37.081252 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:37.081245    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:06:37.097794 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:37.097744    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l" podStartSLOduration=2.09772749 podStartE2EDuration="2.09772749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:06:37.093606523 +0000 UTC m=+1077.344742567" watchObservedRunningTime="2026-09-10 00:06:37.09772749 +0000 UTC m=+1077.348863532"
Sep 10 00:06:48.088882 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:48.088841    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:06:48.089353 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:48.089190    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:06:48.106342 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:06:48.106295    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5" podStartSLOduration=13.106280549 podStartE2EDuration="13.106280549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:06:37.119917176 +0000 UTC m=+1077.371053217" watchObservedRunningTime="2026-09-10 00:06:48.106280549 +0000 UTC m=+1088.357416591"
Sep 10 00:07:05.447910 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.447876    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"]
Sep 10 00:07:05.448475 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.448127    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l" podUID="ee523b67-1f1e-4a8e-9fce-946d946cc63c" containerName="payload-processing" containerID="cri-o://26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec" gracePeriod=30
Sep 10 00:07:05.456804 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.456777    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"]
Sep 10 00:07:05.456992 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.456959    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5" podUID="28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a" containerName="payload-pre-processing" containerID="cri-o://746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b" gracePeriod=30
Sep 10 00:07:05.715043 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.715013    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:07:05.755414 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.755387    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:07:05.816499 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.816462    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hdfd\" (UniqueName: \"kubernetes.io/projected/ee523b67-1f1e-4a8e-9fce-946d946cc63c-kube-api-access-7hdfd\") pod \"ee523b67-1f1e-4a8e-9fce-946d946cc63c\" (UID: \"ee523b67-1f1e-4a8e-9fce-946d946cc63c\") "
Sep 10 00:07:05.816670 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.816509    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-plugins-config-volume\") pod \"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a\" (UID: \"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a\") "
Sep 10 00:07:05.816670 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.816593    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee523b67-1f1e-4a8e-9fce-946d946cc63c-plugins-config-volume\") pod \"ee523b67-1f1e-4a8e-9fce-946d946cc63c\" (UID: \"ee523b67-1f1e-4a8e-9fce-946d946cc63c\") "
Sep 10 00:07:05.816670 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.816627    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkhxp\" (UniqueName: \"kubernetes.io/projected/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-kube-api-access-nkhxp\") pod \"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a\" (UID: \"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a\") "
Sep 10 00:07:05.816942 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.816918    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a" (UID: "28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:07:05.817042 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.817022    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee523b67-1f1e-4a8e-9fce-946d946cc63c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee523b67-1f1e-4a8e-9fce-946d946cc63c" (UID: "ee523b67-1f1e-4a8e-9fce-946d946cc63c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:07:05.818707 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.818687    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee523b67-1f1e-4a8e-9fce-946d946cc63c-kube-api-access-7hdfd" (OuterVolumeSpecName: "kube-api-access-7hdfd") pod "ee523b67-1f1e-4a8e-9fce-946d946cc63c" (UID: "ee523b67-1f1e-4a8e-9fce-946d946cc63c"). InnerVolumeSpecName "kube-api-access-7hdfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:07:05.818787 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.818720    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-kube-api-access-nkhxp" (OuterVolumeSpecName: "kube-api-access-nkhxp") pod "28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a" (UID: "28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a"). InnerVolumeSpecName "kube-api-access-nkhxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:07:05.917576 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.917537    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee523b67-1f1e-4a8e-9fce-946d946cc63c-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:07:05.917576 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.917572    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkhxp\" (UniqueName: \"kubernetes.io/projected/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-kube-api-access-nkhxp\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:07:05.917576 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.917584    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hdfd\" (UniqueName: \"kubernetes.io/projected/ee523b67-1f1e-4a8e-9fce-946d946cc63c-kube-api-access-7hdfd\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:07:05.917816 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:05.917599    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:07:06.251820 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.251784    2556 generic.go:356] "Generic (PLEG): container finished" podID="ee523b67-1f1e-4a8e-9fce-946d946cc63c" containerID="26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec" exitCode=0
Sep 10 00:07:06.252007 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.251844    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"
Sep 10 00:07:06.252007 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.251873    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l" event={"ID":"ee523b67-1f1e-4a8e-9fce-946d946cc63c","Type":"ContainerDied","Data":"26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec"}
Sep 10 00:07:06.252007 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.251923    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l" event={"ID":"ee523b67-1f1e-4a8e-9fce-946d946cc63c","Type":"ContainerDied","Data":"fc61fa843b391021e15855ebe89fcba452ff36da159d91ecf190fa2a1ab26231"}
Sep 10 00:07:06.252007 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.251963    2556 scope.go:117] "RemoveContainer" containerID="26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec"
Sep 10 00:07:06.253376 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.253355    2556 generic.go:356] "Generic (PLEG): container finished" podID="28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a" containerID="746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b" exitCode=0
Sep 10 00:07:06.253481 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.253395    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"
Sep 10 00:07:06.253481 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.253412    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5" event={"ID":"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a","Type":"ContainerDied","Data":"746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b"}
Sep 10 00:07:06.253481 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.253446    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5" event={"ID":"28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a","Type":"ContainerDied","Data":"eac301fd1e8ee28cad18108dfc1c6feb50c2771011a753148b7dbd8e09f62f55"}
Sep 10 00:07:06.266510 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.266491    2556 scope.go:117] "RemoveContainer" containerID="26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec"
Sep 10 00:07:06.266824 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:07:06.266781    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec\": container with ID starting with 26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec not found: ID does not exist" containerID="26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec"
Sep 10 00:07:06.266916 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.266818    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec"} err="failed to get container status \"26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec\": rpc error: code = NotFound desc = could not find container \"26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec\": container with ID starting with 26f6cad22f0f0c1afe5b627660fe4fc1ec12675ea8dd5d5006f646c0e5a3f9ec not found: ID does not exist"
Sep 10 00:07:06.266916 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.266841    2556 scope.go:117] "RemoveContainer" containerID="746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b"
Sep 10 00:07:06.276218 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.276200    2556 scope.go:117] "RemoveContainer" containerID="746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b"
Sep 10 00:07:06.276304 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.276260    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"]
Sep 10 00:07:06.276529 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:07:06.276508    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b\": container with ID starting with 746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b not found: ID does not exist" containerID="746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b"
Sep 10 00:07:06.276588 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.276535    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b"} err="failed to get container status \"746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b\": rpc error: code = NotFound desc = could not find container \"746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b\": container with ID starting with 746e8c5b1c777e8ecf174c06535043796c3e036ddd91a019f4984e18263aaf1b not found: ID does not exist"
Sep 10 00:07:06.278408 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.278387    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-96c64491-6d6c447889-wxt7l"]
Sep 10 00:07:06.286621 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.286602    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"]
Sep 10 00:07:06.290514 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:06.290493    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-96c64491-7fcc6b76cd-g8lf5"]
Sep 10 00:07:08.270408 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.270375    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a" path="/var/lib/kubelet/pods/28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a/volumes"
Sep 10 00:07:08.270772 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.270763    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee523b67-1f1e-4a8e-9fce-946d946cc63c" path="/var/lib/kubelet/pods/ee523b67-1f1e-4a8e-9fce-946d946cc63c/volumes"
Sep 10 00:07:08.271014 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.270999    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"]
Sep 10 00:07:08.271417 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.271405    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a" containerName="payload-pre-processing"
Sep 10 00:07:08.271459 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.271419    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a" containerName="payload-pre-processing"
Sep 10 00:07:08.271459 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.271447    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee523b67-1f1e-4a8e-9fce-946d946cc63c" containerName="payload-processing"
Sep 10 00:07:08.271459 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.271453    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee523b67-1f1e-4a8e-9fce-946d946cc63c" containerName="payload-processing"
Sep 10 00:07:08.271552 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.271517    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="28dbb6f6-2815-40c9-ba1c-10c0ce1bf59a" containerName="payload-pre-processing"
Sep 10 00:07:08.271552 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.271525    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee523b67-1f1e-4a8e-9fce-946d946cc63c" containerName="payload-processing"
Sep 10 00:07:08.274860 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.274844    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:07:08.277318 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.277290    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-8376bc\""
Sep 10 00:07:08.277429 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.277378    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-8376bc-dockercfg-h9grd\""
Sep 10 00:07:08.286310 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.286290    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"]
Sep 10 00:07:08.341669 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.341636    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62dc0c11-dedc-4668-9162-b573bc459900-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8376bc-779c757f85-k45jf\" (UID: \"62dc0c11-dedc-4668-9162-b573bc459900\") " pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:07:08.341805 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.341676    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlwn\" (UniqueName: \"kubernetes.io/projected/62dc0c11-dedc-4668-9162-b573bc459900-kube-api-access-mhlwn\") pod \"payload-processing-e2e-explicit-8376bc-779c757f85-k45jf\" (UID: \"62dc0c11-dedc-4668-9162-b573bc459900\") " pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:07:08.442972 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.442945    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62dc0c11-dedc-4668-9162-b573bc459900-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8376bc-779c757f85-k45jf\" (UID: \"62dc0c11-dedc-4668-9162-b573bc459900\") " pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:07:08.443132 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.442978    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlwn\" (UniqueName: \"kubernetes.io/projected/62dc0c11-dedc-4668-9162-b573bc459900-kube-api-access-mhlwn\") pod \"payload-processing-e2e-explicit-8376bc-779c757f85-k45jf\" (UID: \"62dc0c11-dedc-4668-9162-b573bc459900\") " pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:07:08.443623 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.443603    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62dc0c11-dedc-4668-9162-b573bc459900-plugins-config-volume\") pod \"payload-processing-e2e-explicit-8376bc-779c757f85-k45jf\" (UID: \"62dc0c11-dedc-4668-9162-b573bc459900\") " pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:07:08.450559 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.450528    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlwn\" (UniqueName: \"kubernetes.io/projected/62dc0c11-dedc-4668-9162-b573bc459900-kube-api-access-mhlwn\") pod \"payload-processing-e2e-explicit-8376bc-779c757f85-k45jf\" (UID: \"62dc0c11-dedc-4668-9162-b573bc459900\") " pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:07:08.586196 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.586123    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:07:08.855222 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:08.855193    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"]
Sep 10 00:07:08.856580 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:07:08.856552    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dc0c11_dedc_4668_9162_b573bc459900.slice/crio-b27bde8444a28b3b84c4162a6097b30357bc8e7290d536675c96ad093e6bbfb9 WatchSource:0}: Error finding container b27bde8444a28b3b84c4162a6097b30357bc8e7290d536675c96ad093e6bbfb9: Status 404 returned error can't find the container with id b27bde8444a28b3b84c4162a6097b30357bc8e7290d536675c96ad093e6bbfb9
Sep 10 00:07:09.276440 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:09.276403    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf" event={"ID":"62dc0c11-dedc-4668-9162-b573bc459900","Type":"ContainerStarted","Data":"26723ce2a7f6289c4537df429c78f03351c29647217158e33a5675ca2b3bc9a8"}
Sep 10 00:07:09.276960 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:09.276464    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf" event={"ID":"62dc0c11-dedc-4668-9162-b573bc459900","Type":"ContainerStarted","Data":"b27bde8444a28b3b84c4162a6097b30357bc8e7290d536675c96ad093e6bbfb9"}
Sep 10 00:07:09.276960 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:09.276496    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:07:09.292206 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:09.292152    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf" podStartSLOduration=1.29213369 podStartE2EDuration="1.29213369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:07:09.29051175 +0000 UTC m=+1109.541647793" watchObservedRunningTime="2026-09-10 00:07:09.29213369 +0000 UTC m=+1109.543269792"
Sep 10 00:07:20.283579 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:20.283552    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:07:24.155222 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.155168    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"]
Sep 10 00:07:24.159643 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.159618    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.161838 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.161818    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"premium-w0-01b7d5-kserve-self-signed-certs\""
Sep 10 00:07:24.168638 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.168613    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"]
Sep 10 00:07:24.282149 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.282115    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfx2b\" (UniqueName: \"kubernetes.io/projected/2993af8b-6a57-4b6c-8046-48297c8323a4-kube-api-access-xfx2b\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.282301 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.282235    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-tmp-dir\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.282301 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.282264    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-dshm\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.282301 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.282290    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-model-cache\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.282440 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.282304    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2993af8b-6a57-4b6c-8046-48297c8323a4-tls-certs\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.282440 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.282321    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-home\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.383699 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.383671    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-tmp-dir\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.383699 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.383700    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-dshm\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.383965 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.383724    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-model-cache\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.383965 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.383747    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2993af8b-6a57-4b6c-8046-48297c8323a4-tls-certs\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.383965 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.383770    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-home\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.383965 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.383845    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfx2b\" (UniqueName: \"kubernetes.io/projected/2993af8b-6a57-4b6c-8046-48297c8323a4-kube-api-access-xfx2b\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.384208 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.384092    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-model-cache\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.384208 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.384153    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-tmp-dir\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.384290 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.384236    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-home\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.386094 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.386073    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-dshm\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.386276 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.386257    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2993af8b-6a57-4b6c-8046-48297c8323a4-tls-certs\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.391289 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.391265    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfx2b\" (UniqueName: \"kubernetes.io/projected/2993af8b-6a57-4b6c-8046-48297c8323a4-kube-api-access-xfx2b\") pod \"premium-w0-01b7d5-kserve-85fdf564bf-tmzxx\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") " pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.469714 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.469687    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:24.739151 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:24.739127    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"]
Sep 10 00:07:24.744845 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:07:24.744807    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2993af8b_6a57_4b6c_8046_48297c8323a4.slice/crio-fc7a34aea5760549238b53a1640140179ea3afbdb6a9cb4c7922356db4725140 WatchSource:0}: Error finding container fc7a34aea5760549238b53a1640140179ea3afbdb6a9cb4c7922356db4725140: Status 404 returned error can't find the container with id fc7a34aea5760549238b53a1640140179ea3afbdb6a9cb4c7922356db4725140
Sep 10 00:07:25.359284 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:25.359248    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx" event={"ID":"2993af8b-6a57-4b6c-8046-48297c8323a4","Type":"ContainerStarted","Data":"6a64fe5098a4aa80e3a3e189d2eb21e8e639a681ad00b1a0a898b922d029ce99"}
Sep 10 00:07:25.359658 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:25.359292    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx" event={"ID":"2993af8b-6a57-4b6c-8046-48297c8323a4","Type":"ContainerStarted","Data":"fc7a34aea5760549238b53a1640140179ea3afbdb6a9cb4c7922356db4725140"}
Sep 10 00:07:25.377422 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:25.377375    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx" podStartSLOduration=1.377360232 podStartE2EDuration="1.377360232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:07:25.376012009 +0000 UTC m=+1125.627148053" watchObservedRunningTime="2026-09-10 00:07:25.377360232 +0000 UTC m=+1125.628496274"
Sep 10 00:07:34.469812 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:34.469779    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:34.470366 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:34.470047    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:34.482454 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:34.482432    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:07:35.421352 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:07:35.421320    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:08:34.464491 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.464410    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"]
Sep 10 00:08:34.468344 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.468326    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.470724 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.470701    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"distinct-w0-01b7d5-kserve-self-signed-certs\""
Sep 10 00:08:34.477220 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.477198    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"]
Sep 10 00:08:34.507531 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.507505    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tmp-dir\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.507646 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.507545    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-home\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.507646 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.507576    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl88m\" (UniqueName: \"kubernetes.io/projected/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-kube-api-access-jl88m\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.507755 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.507651    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-model-cache\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.507755 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.507681    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-dshm\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.507755 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.507705    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tls-certs\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.609019 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.608992    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-home\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.609147 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.609029    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl88m\" (UniqueName: \"kubernetes.io/projected/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-kube-api-access-jl88m\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.609147 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.609075    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-model-cache\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.609147 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.609100    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-dshm\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.609147 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.609120    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tls-certs\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.609316 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.609207    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tmp-dir\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.609469 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.609449    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-home\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.609517 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.609488    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-model-cache\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.609601 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.609585    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tmp-dir\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.611305 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.611284    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-dshm\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.611548 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.611530    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tls-certs\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.616743 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.616716    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl88m\" (UniqueName: \"kubernetes.io/projected/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-kube-api-access-jl88m\") pod \"distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") " pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:34.779691 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:34.779638    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:35.048638 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:35.048555    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"]
Sep 10 00:08:35.051960 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:08:35.051930    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f455ae5_9cdb_4a87_834b_4f1bb4c499d1.slice/crio-951605e8e1981aed539c6a9d6ec9f471c009c1d1e8a6cab77afa14ed2ed22832 WatchSource:0}: Error finding container 951605e8e1981aed539c6a9d6ec9f471c009c1d1e8a6cab77afa14ed2ed22832: Status 404 returned error can't find the container with id 951605e8e1981aed539c6a9d6ec9f471c009c1d1e8a6cab77afa14ed2ed22832
Sep 10 00:08:35.734281 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:35.734247    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb" event={"ID":"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1","Type":"ContainerStarted","Data":"267e6f543aa7d7a8257cd87328af61307f05bae6e1f81cca3f541b25c0dbc9a1"}
Sep 10 00:08:35.734281 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:35.734292    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb" event={"ID":"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1","Type":"ContainerStarted","Data":"951605e8e1981aed539c6a9d6ec9f471c009c1d1e8a6cab77afa14ed2ed22832"}
Sep 10 00:08:35.751672 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:35.751628    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb" podStartSLOduration=1.751608748 podStartE2EDuration="1.751608748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:08:35.749159145 +0000 UTC m=+1196.000295189" watchObservedRunningTime="2026-09-10 00:08:35.751608748 +0000 UTC m=+1196.002744796"
Sep 10 00:08:43.661226 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:43.661194    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"]
Sep 10 00:08:43.661672 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:43.661439    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf" podUID="62dc0c11-dedc-4668-9162-b573bc459900" containerName="payload-processing" containerID="cri-o://26723ce2a7f6289c4537df429c78f03351c29647217158e33a5675ca2b3bc9a8" gracePeriod=30
Sep 10 00:08:43.783461 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:43.783427    2556 generic.go:356] "Generic (PLEG): container finished" podID="62dc0c11-dedc-4668-9162-b573bc459900" containerID="26723ce2a7f6289c4537df429c78f03351c29647217158e33a5675ca2b3bc9a8" exitCode=0
Sep 10 00:08:43.783593 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:43.783493    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf" event={"ID":"62dc0c11-dedc-4668-9162-b573bc459900","Type":"ContainerDied","Data":"26723ce2a7f6289c4537df429c78f03351c29647217158e33a5675ca2b3bc9a8"}
Sep 10 00:08:43.915905 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:43.915845    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:08:43.985241 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:43.985209    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62dc0c11-dedc-4668-9162-b573bc459900-plugins-config-volume\") pod \"62dc0c11-dedc-4668-9162-b573bc459900\" (UID: \"62dc0c11-dedc-4668-9162-b573bc459900\") "
Sep 10 00:08:43.985378 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:43.985287    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhlwn\" (UniqueName: \"kubernetes.io/projected/62dc0c11-dedc-4668-9162-b573bc459900-kube-api-access-mhlwn\") pod \"62dc0c11-dedc-4668-9162-b573bc459900\" (UID: \"62dc0c11-dedc-4668-9162-b573bc459900\") "
Sep 10 00:08:43.985629 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:43.985601    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dc0c11-dedc-4668-9162-b573bc459900-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62dc0c11-dedc-4668-9162-b573bc459900" (UID: "62dc0c11-dedc-4668-9162-b573bc459900"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:08:43.987279 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:43.987253    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dc0c11-dedc-4668-9162-b573bc459900-kube-api-access-mhlwn" (OuterVolumeSpecName: "kube-api-access-mhlwn") pod "62dc0c11-dedc-4668-9162-b573bc459900" (UID: "62dc0c11-dedc-4668-9162-b573bc459900"). InnerVolumeSpecName "kube-api-access-mhlwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:08:44.086889 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:44.086864    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhlwn\" (UniqueName: \"kubernetes.io/projected/62dc0c11-dedc-4668-9162-b573bc459900-kube-api-access-mhlwn\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:08:44.086889 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:44.086887    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62dc0c11-dedc-4668-9162-b573bc459900-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:08:44.780672 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:44.780639    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:44.781047 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:44.780899    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:44.791356 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:44.791338    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"
Sep 10 00:08:44.791468 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:44.791374    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf" event={"ID":"62dc0c11-dedc-4668-9162-b573bc459900","Type":"ContainerDied","Data":"b27bde8444a28b3b84c4162a6097b30357bc8e7290d536675c96ad093e6bbfb9"}
Sep 10 00:08:44.791468 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:44.791422    2556 scope.go:117] "RemoveContainer" containerID="26723ce2a7f6289c4537df429c78f03351c29647217158e33a5675ca2b3bc9a8"
Sep 10 00:08:44.794811 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:44.794792    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:44.807189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:44.807164    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"]
Sep 10 00:08:44.809233 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:44.809212    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-8376bc-779c757f85-k45jf"]
Sep 10 00:08:45.808967 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:45.808933    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:08:46.268500 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:46.268463    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62dc0c11-dedc-4668-9162-b573bc459900" path="/var/lib/kubelet/pods/62dc0c11-dedc-4668-9162-b573bc459900/volumes"
Sep 10 00:08:49.113393 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.113356    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"]
Sep 10 00:08:49.113806 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.113793    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dc0c11-dedc-4668-9162-b573bc459900" containerName="payload-processing"
Sep 10 00:08:49.113806 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.113807    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dc0c11-dedc-4668-9162-b573bc459900" containerName="payload-processing"
Sep 10 00:08:49.113898 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.113889    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="62dc0c11-dedc-4668-9162-b573bc459900" containerName="payload-processing"
Sep 10 00:08:49.118044 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.118027    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:08:49.120441 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.120412    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cb0fe265\""
Sep 10 00:08:49.120562 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.120472    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cb0fe265-dockercfg-d8flb\""
Sep 10 00:08:49.129912 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.129876    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"]
Sep 10 00:08:49.234826 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.234796    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc\" (UID: \"a67dfd25-c719-40a5-bb79-ba2b7a41baf3\") " pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:08:49.234995 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.234974    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrxp2\" (UniqueName: \"kubernetes.io/projected/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-kube-api-access-hrxp2\") pod \"payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc\" (UID: \"a67dfd25-c719-40a5-bb79-ba2b7a41baf3\") " pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:08:49.335519 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.335487    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrxp2\" (UniqueName: \"kubernetes.io/projected/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-kube-api-access-hrxp2\") pod \"payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc\" (UID: \"a67dfd25-c719-40a5-bb79-ba2b7a41baf3\") " pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:08:49.335675 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.335531    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc\" (UID: \"a67dfd25-c719-40a5-bb79-ba2b7a41baf3\") " pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:08:49.336233 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.336213    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc\" (UID: \"a67dfd25-c719-40a5-bb79-ba2b7a41baf3\") " pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:08:49.343118 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.343092    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrxp2\" (UniqueName: \"kubernetes.io/projected/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-kube-api-access-hrxp2\") pod \"payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc\" (UID: \"a67dfd25-c719-40a5-bb79-ba2b7a41baf3\") " pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:08:49.430871 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.430814    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:08:49.700447 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.700416    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"]
Sep 10 00:08:49.701439 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:08:49.701411    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67dfd25_c719_40a5_bb79_ba2b7a41baf3.slice/crio-948bc1f4e677742d0f1da73f98455f29dd6ffb228ee243b31a8ce181553e2739 WatchSource:0}: Error finding container 948bc1f4e677742d0f1da73f98455f29dd6ffb228ee243b31a8ce181553e2739: Status 404 returned error can't find the container with id 948bc1f4e677742d0f1da73f98455f29dd6ffb228ee243b31a8ce181553e2739
Sep 10 00:08:49.820807 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.820780    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc" event={"ID":"a67dfd25-c719-40a5-bb79-ba2b7a41baf3","Type":"ContainerStarted","Data":"ca7fb901d50bbd09a18da73c2a0e62b9311cde8af4cc230c2b6cc708aa063f7c"}
Sep 10 00:08:49.820939 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.820836    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc" event={"ID":"a67dfd25-c719-40a5-bb79-ba2b7a41baf3","Type":"ContainerStarted","Data":"948bc1f4e677742d0f1da73f98455f29dd6ffb228ee243b31a8ce181553e2739"}
Sep 10 00:08:49.820939 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.820866    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:08:49.837657 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:08:49.837615    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc" podStartSLOduration=0.837599359 podStartE2EDuration="837.599359ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:08:49.83525121 +0000 UTC m=+1210.086387253" watchObservedRunningTime="2026-09-10 00:08:49.837599359 +0000 UTC m=+1210.088735401"
Sep 10 00:09:00.828733 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:00.828698    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:09:19.205512 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:19.205478    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"]
Sep 10 00:09:19.209226 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:19.209184    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"
Sep 10 00:09:19.211431 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:19.211407    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:09:19.211578 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:19.211437    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:09:19.216253 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:19.216226    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"]
Sep 10 00:09:19.297287 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:19.297254    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0572f163-29ff-462b-96ea-aca3bc863941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5\" (UID: \"0572f163-29ff-462b-96ea-aca3bc863941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"
Sep 10 00:09:19.398357 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:19.398313    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0572f163-29ff-462b-96ea-aca3bc863941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5\" (UID: \"0572f163-29ff-462b-96ea-aca3bc863941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"
Sep 10 00:09:19.398892 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:19.398870    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0572f163-29ff-462b-96ea-aca3bc863941-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5\" (UID: \"0572f163-29ff-462b-96ea-aca3bc863941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"
Sep 10 00:09:19.521510 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:19.521442    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"
Sep 10 00:09:19.792251 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:19.792227    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"]
Sep 10 00:09:19.794380 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:09:19.794342    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0572f163_29ff_462b_96ea_aca3bc863941.slice/crio-da041ec19309a2413fcd563180ef9c326f5c76351f3ef03690c885ac0b201013 WatchSource:0}: Error finding container da041ec19309a2413fcd563180ef9c326f5c76351f3ef03690c885ac0b201013: Status 404 returned error can't find the container with id da041ec19309a2413fcd563180ef9c326f5c76351f3ef03690c885ac0b201013
Sep 10 00:09:19.992045 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:19.992011    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5" event={"ID":"0572f163-29ff-462b-96ea-aca3bc863941","Type":"ContainerStarted","Data":"da041ec19309a2413fcd563180ef9c326f5c76351f3ef03690c885ac0b201013"}
Sep 10 00:09:20.998786 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:20.998752    2556 generic.go:356] "Generic (PLEG): container finished" podID="0572f163-29ff-462b-96ea-aca3bc863941" containerID="beac0204d9449f0e3bd36e468bb5ca90a1fd5b8cd535111d7deec40f1d2d3d25" exitCode=0
Sep 10 00:09:20.999273 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:20.998835    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5" event={"ID":"0572f163-29ff-462b-96ea-aca3bc863941","Type":"ContainerDied","Data":"beac0204d9449f0e3bd36e468bb5ca90a1fd5b8cd535111d7deec40f1d2d3d25"}
Sep 10 00:09:22.145667 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:22.145643    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"
Sep 10 00:09:22.222257 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:22.222236    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0572f163-29ff-462b-96ea-aca3bc863941-maas-api-service-ca\") pod \"0572f163-29ff-462b-96ea-aca3bc863941\" (UID: \"0572f163-29ff-462b-96ea-aca3bc863941\") "
Sep 10 00:09:22.222563 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:22.222543    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0572f163-29ff-462b-96ea-aca3bc863941-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0572f163-29ff-462b-96ea-aca3bc863941" (UID: "0572f163-29ff-462b-96ea-aca3bc863941"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:09:22.324512 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:22.324490    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0572f163-29ff-462b-96ea-aca3bc863941-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:09:23.011757 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:23.011728    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"
Sep 10 00:09:23.011757 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:23.011746    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5" event={"ID":"0572f163-29ff-462b-96ea-aca3bc863941","Type":"ContainerDied","Data":"da041ec19309a2413fcd563180ef9c326f5c76351f3ef03690c885ac0b201013"}
Sep 10 00:09:23.011961 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:23.011812    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da041ec19309a2413fcd563180ef9c326f5c76351f3ef03690c885ac0b201013"
Sep 10 00:09:29.927276 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:29.927236    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"]
Sep 10 00:09:29.927835 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:29.927460    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc" podUID="a67dfd25-c719-40a5-bb79-ba2b7a41baf3" containerName="payload-processing" containerID="cri-o://ca7fb901d50bbd09a18da73c2a0e62b9311cde8af4cc230c2b6cc708aa063f7c" gracePeriod=30
Sep 10 00:09:30.052260 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.052229    2556 generic.go:356] "Generic (PLEG): container finished" podID="a67dfd25-c719-40a5-bb79-ba2b7a41baf3" containerID="ca7fb901d50bbd09a18da73c2a0e62b9311cde8af4cc230c2b6cc708aa063f7c" exitCode=0
Sep 10 00:09:30.052402 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.052299    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc" event={"ID":"a67dfd25-c719-40a5-bb79-ba2b7a41baf3","Type":"ContainerDied","Data":"ca7fb901d50bbd09a18da73c2a0e62b9311cde8af4cc230c2b6cc708aa063f7c"}
Sep 10 00:09:30.076271 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.076224    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"]
Sep 10 00:09:30.078574 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.078551    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb0fe265-qwzv5"]
Sep 10 00:09:30.198141 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.198112    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:09:30.268128 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.268101    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0572f163-29ff-462b-96ea-aca3bc863941" path="/var/lib/kubelet/pods/0572f163-29ff-462b-96ea-aca3bc863941/volumes"
Sep 10 00:09:30.296560 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.296537    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-plugins-config-volume\") pod \"a67dfd25-c719-40a5-bb79-ba2b7a41baf3\" (UID: \"a67dfd25-c719-40a5-bb79-ba2b7a41baf3\") "
Sep 10 00:09:30.296672 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.296567    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrxp2\" (UniqueName: \"kubernetes.io/projected/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-kube-api-access-hrxp2\") pod \"a67dfd25-c719-40a5-bb79-ba2b7a41baf3\" (UID: \"a67dfd25-c719-40a5-bb79-ba2b7a41baf3\") "
Sep 10 00:09:30.296856 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.296833    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a67dfd25-c719-40a5-bb79-ba2b7a41baf3" (UID: "a67dfd25-c719-40a5-bb79-ba2b7a41baf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:09:30.297040 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.297014    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:09:30.298615 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.298595    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-kube-api-access-hrxp2" (OuterVolumeSpecName: "kube-api-access-hrxp2") pod "a67dfd25-c719-40a5-bb79-ba2b7a41baf3" (UID: "a67dfd25-c719-40a5-bb79-ba2b7a41baf3"). InnerVolumeSpecName "kube-api-access-hrxp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:09:30.398028 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:30.398007    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrxp2\" (UniqueName: \"kubernetes.io/projected/a67dfd25-c719-40a5-bb79-ba2b7a41baf3-kube-api-access-hrxp2\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:09:31.059669 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:31.059642    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"
Sep 10 00:09:31.060030 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:31.059690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc" event={"ID":"a67dfd25-c719-40a5-bb79-ba2b7a41baf3","Type":"ContainerDied","Data":"948bc1f4e677742d0f1da73f98455f29dd6ffb228ee243b31a8ce181553e2739"}
Sep 10 00:09:31.060030 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:31.059751    2556 scope.go:117] "RemoveContainer" containerID="ca7fb901d50bbd09a18da73c2a0e62b9311cde8af4cc230c2b6cc708aa063f7c"
Sep 10 00:09:31.080303 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:31.080276    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"]
Sep 10 00:09:31.083649 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:31.083627    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb0fe265-689f768665-vcwmc"]
Sep 10 00:09:32.268295 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:32.268259    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a67dfd25-c719-40a5-bb79-ba2b7a41baf3" path="/var/lib/kubelet/pods/a67dfd25-c719-40a5-bb79-ba2b7a41baf3/volumes"
Sep 10 00:09:35.687816 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.687782    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"]
Sep 10 00:09:35.688260 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.688245    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a67dfd25-c719-40a5-bb79-ba2b7a41baf3" containerName="payload-processing"
Sep 10 00:09:35.688307 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.688260    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a67dfd25-c719-40a5-bb79-ba2b7a41baf3" containerName="payload-processing"
Sep 10 00:09:35.688307 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.688278    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0572f163-29ff-462b-96ea-aca3bc863941" containerName="revoke-keys"
Sep 10 00:09:35.688307 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.688285    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0572f163-29ff-462b-96ea-aca3bc863941" containerName="revoke-keys"
Sep 10 00:09:35.688406 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.688346    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a67dfd25-c719-40a5-bb79-ba2b7a41baf3" containerName="payload-processing"
Sep 10 00:09:35.688406 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.688354    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0572f163-29ff-462b-96ea-aca3bc863941" containerName="revoke-keys"
Sep 10 00:09:35.692688 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.692673    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.695834 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.695812    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c59d42\"/\"kube-root-ca.crt\""
Sep 10 00:09:35.695918 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.695897    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c59d42\"/\"noten-model-c59d42-kserve-self-signed-certs\""
Sep 10 00:09:35.695971 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.695909    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c59d42\"/\"openshift-service-ca.crt\""
Sep 10 00:09:35.695971 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.695963    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c59d42\"/\"default-dockercfg-p975h\""
Sep 10 00:09:35.701035 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.701014    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"]
Sep 10 00:09:35.742544 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.742521    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-dshm\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.742649 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.742575    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tls-certs\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.742689 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.742641    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-model-cache\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.742739 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.742720    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447k9\" (UniqueName: \"kubernetes.io/projected/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-kube-api-access-447k9\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.742776 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.742763    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tmp-dir\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.742814 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.742805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-home\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.843978 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.843951    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-model-cache\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.844148 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.843992    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447k9\" (UniqueName: \"kubernetes.io/projected/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-kube-api-access-447k9\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.844148 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.844016    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tmp-dir\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.844148 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.844046    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-home\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.844148 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.844075    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-dshm\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.844148 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.844134    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tls-certs\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.844456 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.844428    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-model-cache\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.844584 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.844478    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tmp-dir\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.844584 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.844500    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-home\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.846340 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.846318    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-dshm\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.846503 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.846487    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tls-certs\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:35.850977 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:35.850956    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-447k9\" (UniqueName: \"kubernetes.io/projected/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-kube-api-access-447k9\") pod \"noten-model-c59d42-kserve-687bbfbf6c-58gfw\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") " pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:36.002742 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:36.002717    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:36.278298 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:36.278270    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"]
Sep 10 00:09:36.279698 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:09:36.279671    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bf1aff8_3204_4b86_a2cc_3440ec86d9f5.slice/crio-e32f3fb06e3625034cf1c31c64b32aeffa7ba95ccbf732745c8143321185770c WatchSource:0}: Error finding container e32f3fb06e3625034cf1c31c64b32aeffa7ba95ccbf732745c8143321185770c: Status 404 returned error can't find the container with id e32f3fb06e3625034cf1c31c64b32aeffa7ba95ccbf732745c8143321185770c
Sep 10 00:09:37.097406 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:37.097367    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw" event={"ID":"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5","Type":"ContainerStarted","Data":"c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38"}
Sep 10 00:09:37.097853 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:37.097416    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw" event={"ID":"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5","Type":"ContainerStarted","Data":"e32f3fb06e3625034cf1c31c64b32aeffa7ba95ccbf732745c8143321185770c"}
Sep 10 00:09:37.112105 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:37.112041    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw" podStartSLOduration=2.112022239 podStartE2EDuration="2.112022239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:09:37.111900627 +0000 UTC m=+1257.363036673" watchObservedRunningTime="2026-09-10 00:09:37.112022239 +0000 UTC m=+1257.363158284"
Sep 10 00:09:46.003557 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:46.003518    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:46.003929 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:46.003599    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:46.015893 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:46.015864    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:46.157684 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:46.157660    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:09:53.650931 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:09:53.650897    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"]
Sep 10 00:10:08.673458 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.673397    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh" podUID="6a424378-3af5-434e-a810-5621681078c4" containerName="main" containerID="cri-o://9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d" gracePeriod=45
Sep 10 00:10:08.919950 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.919925    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:10:08.952486 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.952464    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-tmp-dir\") pod \"6a424378-3af5-434e-a810-5621681078c4\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") "
Sep 10 00:10:08.952609 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.952512    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-model-cache\") pod \"6a424378-3af5-434e-a810-5621681078c4\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") "
Sep 10 00:10:08.952609 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.952561    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-dshm\") pod \"6a424378-3af5-434e-a810-5621681078c4\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") "
Sep 10 00:10:08.952609 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.952584    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-home\") pod \"6a424378-3af5-434e-a810-5621681078c4\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") "
Sep 10 00:10:08.952774 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.952624    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qxsz\" (UniqueName: \"kubernetes.io/projected/6a424378-3af5-434e-a810-5621681078c4-kube-api-access-5qxsz\") pod \"6a424378-3af5-434e-a810-5621681078c4\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") "
Sep 10 00:10:08.952774 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.952651    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a424378-3af5-434e-a810-5621681078c4-tls-certs\") pod \"6a424378-3af5-434e-a810-5621681078c4\" (UID: \"6a424378-3af5-434e-a810-5621681078c4\") "
Sep 10 00:10:08.952774 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.952744    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a424378-3af5-434e-a810-5621681078c4" (UID: "6a424378-3af5-434e-a810-5621681078c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:08.952949 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.952891    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a424378-3af5-434e-a810-5621681078c4" (UID: "6a424378-3af5-434e-a810-5621681078c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:08.953023 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.952953    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-home" (OuterVolumeSpecName: "home") pod "6a424378-3af5-434e-a810-5621681078c4" (UID: "6a424378-3af5-434e-a810-5621681078c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:08.953123 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.953107    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:08.953181 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.953129    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:08.953181 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.953142    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:08.955104 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.955072    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a424378-3af5-434e-a810-5621681078c4-kube-api-access-5qxsz" (OuterVolumeSpecName: "kube-api-access-5qxsz") pod "6a424378-3af5-434e-a810-5621681078c4" (UID: "6a424378-3af5-434e-a810-5621681078c4"). InnerVolumeSpecName "kube-api-access-5qxsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:10:08.955104 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.955090    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-dshm" (OuterVolumeSpecName: "dshm") pod "6a424378-3af5-434e-a810-5621681078c4" (UID: "6a424378-3af5-434e-a810-5621681078c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:08.955254 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:08.955204    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a424378-3af5-434e-a810-5621681078c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a424378-3af5-434e-a810-5621681078c4" (UID: "6a424378-3af5-434e-a810-5621681078c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:10:09.054159 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.054128    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a424378-3af5-434e-a810-5621681078c4-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:09.054159 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.054160    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qxsz\" (UniqueName: \"kubernetes.io/projected/6a424378-3af5-434e-a810-5621681078c4-kube-api-access-5qxsz\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:09.054322 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.054172    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a424378-3af5-434e-a810-5621681078c4-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:09.272005 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.271926    2556 generic.go:356] "Generic (PLEG): container finished" podID="6a424378-3af5-434e-a810-5621681078c4" containerID="9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d" exitCode=0
Sep 10 00:10:09.272005 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.271966    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"
Sep 10 00:10:09.272237 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.272008    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh" event={"ID":"6a424378-3af5-434e-a810-5621681078c4","Type":"ContainerDied","Data":"9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d"}
Sep 10 00:10:09.272237 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.272043    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh" event={"ID":"6a424378-3af5-434e-a810-5621681078c4","Type":"ContainerDied","Data":"6195a910888b575822fd78e62043e4803b53ab843534966736ee7919e544de59"}
Sep 10 00:10:09.272237 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.272083    2556 scope.go:117] "RemoveContainer" containerID="9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d"
Sep 10 00:10:09.281482 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.281456    2556 scope.go:117] "RemoveContainer" containerID="9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d"
Sep 10 00:10:09.281767 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:10:09.281743    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d\": container with ID starting with 9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d not found: ID does not exist" containerID="9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d"
Sep 10 00:10:09.281848 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.281775    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d"} err="failed to get container status \"9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d\": rpc error: code = NotFound desc = could not find container \"9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d\": container with ID starting with 9e2684cf1458a3e8772d01d41dba18452d95e8f3b0e6469deac6d4cedd551c0d not found: ID does not exist"
Sep 10 00:10:09.292770 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.292746    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"]
Sep 10 00:10:09.296276 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:09.296258    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f0f965/test-model-f0f965-kserve-784f76ff69-7xrsh"]
Sep 10 00:10:10.269348 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:10.269314    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a424378-3af5-434e-a810-5621681078c4" path="/var/lib/kubelet/pods/6a424378-3af5-434e-a810-5621681078c4/volumes"
Sep 10 00:10:12.683272 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:12.683237    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"]
Sep 10 00:10:12.683948 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:12.683928    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a424378-3af5-434e-a810-5621681078c4" containerName="main"
Sep 10 00:10:12.684031 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:12.683952    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a424378-3af5-434e-a810-5621681078c4" containerName="main"
Sep 10 00:10:12.684110 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:12.684095    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a424378-3af5-434e-a810-5621681078c4" containerName="main"
Sep 10 00:10:12.687724 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:12.687698    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"]
Sep 10 00:10:12.687870 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:12.687808    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"
Sep 10 00:10:12.690525 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:12.690500    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:10:12.690734 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:12.690503    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:10:12.789409 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:12.789376    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef2ae9a9-41e4-450e-9b49-9297b3ed7afa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh\" (UID: \"ef2ae9a9-41e4-450e-9b49-9297b3ed7afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"
Sep 10 00:10:12.889888 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:12.889841    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef2ae9a9-41e4-450e-9b49-9297b3ed7afa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh\" (UID: \"ef2ae9a9-41e4-450e-9b49-9297b3ed7afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"
Sep 10 00:10:12.890477 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:12.890452    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef2ae9a9-41e4-450e-9b49-9297b3ed7afa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh\" (UID: \"ef2ae9a9-41e4-450e-9b49-9297b3ed7afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"
Sep 10 00:10:13.000476 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:13.000439    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"
Sep 10 00:10:13.272998 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:13.272961    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"]
Sep 10 00:10:13.274815 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:10:13.274787    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2ae9a9_41e4_450e_9b49_9297b3ed7afa.slice/crio-3d5f17c6ac1e1109fc026bd12cca9da01f01e9720f5fcbda44eaf5dc0ce3bbfe WatchSource:0}: Error finding container 3d5f17c6ac1e1109fc026bd12cca9da01f01e9720f5fcbda44eaf5dc0ce3bbfe: Status 404 returned error can't find the container with id 3d5f17c6ac1e1109fc026bd12cca9da01f01e9720f5fcbda44eaf5dc0ce3bbfe
Sep 10 00:10:13.297108 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:13.297078    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh" event={"ID":"ef2ae9a9-41e4-450e-9b49-9297b3ed7afa","Type":"ContainerStarted","Data":"3d5f17c6ac1e1109fc026bd12cca9da01f01e9720f5fcbda44eaf5dc0ce3bbfe"}
Sep 10 00:10:14.303422 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:14.303392    2556 generic.go:356] "Generic (PLEG): container finished" podID="ef2ae9a9-41e4-450e-9b49-9297b3ed7afa" containerID="0e9c41e902ee25abc828f10afdef3ae6eb72507435207dcb057f5ebe43eec4d0" exitCode=0
Sep 10 00:10:14.303808 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:14.303475    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh" event={"ID":"ef2ae9a9-41e4-450e-9b49-9297b3ed7afa","Type":"ContainerDied","Data":"0e9c41e902ee25abc828f10afdef3ae6eb72507435207dcb057f5ebe43eec4d0"}
Sep 10 00:10:15.446793 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.446769    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"
Sep 10 00:10:15.513434 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.513407    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef2ae9a9-41e4-450e-9b49-9297b3ed7afa-maas-api-service-ca\") pod \"ef2ae9a9-41e4-450e-9b49-9297b3ed7afa\" (UID: \"ef2ae9a9-41e4-450e-9b49-9297b3ed7afa\") "
Sep 10 00:10:15.513767 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.513745    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2ae9a9-41e4-450e-9b49-9297b3ed7afa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef2ae9a9-41e4-450e-9b49-9297b3ed7afa" (UID: "ef2ae9a9-41e4-450e-9b49-9297b3ed7afa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:10:15.614317 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.614235    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef2ae9a9-41e4-450e-9b49-9297b3ed7afa-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:15.720199 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.720162    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"]
Sep 10 00:10:15.840806 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.840766    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"]
Sep 10 00:10:15.841533 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.841509    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef2ae9a9-41e4-450e-9b49-9297b3ed7afa" containerName="revoke-keys"
Sep 10 00:10:15.841533 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.841532    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef2ae9a9-41e4-450e-9b49-9297b3ed7afa" containerName="revoke-keys"
Sep 10 00:10:15.841692 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.841640    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef2ae9a9-41e4-450e-9b49-9297b3ed7afa" containerName="revoke-keys"
Sep 10 00:10:15.844864 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.844841    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:15.847588 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.847557    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc0040\"/\"default-dockercfg-jt2wp\""
Sep 10 00:10:15.847948 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.847617    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc0040\"/\"kube-root-ca.crt\""
Sep 10 00:10:15.847948 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.847739    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc0040\"/\"distinct-2-w4-cc0040-kserve-self-signed-certs\""
Sep 10 00:10:15.847948 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.847825    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc0040\"/\"openshift-service-ca.crt\""
Sep 10 00:10:15.854734 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.854715    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"]
Sep 10 00:10:15.916905 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.916820    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tls-certs\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:15.916905 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.916865    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-model-cache\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:15.917199 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.916980    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-home\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:15.917199 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.917029    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-dshm\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:15.917199 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.917083    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tmp-dir\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:15.917199 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:15.917165    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khghd\" (UniqueName: \"kubernetes.io/projected/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-kube-api-access-khghd\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.017796 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.017762    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-dshm\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.017796 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.017796    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tmp-dir\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.017995 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.017843    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khghd\" (UniqueName: \"kubernetes.io/projected/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-kube-api-access-khghd\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.017995 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.017967    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tls-certs\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.018124 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.018047    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-model-cache\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.018218 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.018177    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-home\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.018360 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.018315    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tmp-dir\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.018424 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.018407    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-model-cache\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.018496 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.018481    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-home\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.019853 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.019834    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-dshm\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.020719 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.020697    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tls-certs\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.029573 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.029545    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khghd\" (UniqueName: \"kubernetes.io/projected/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-kube-api-access-khghd\") pod \"distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") " pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.156265 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.156231    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:16.315603 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.315571    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"
Sep 10 00:10:16.315769 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.315613    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh" event={"ID":"ef2ae9a9-41e4-450e-9b49-9297b3ed7afa","Type":"ContainerDied","Data":"3d5f17c6ac1e1109fc026bd12cca9da01f01e9720f5fcbda44eaf5dc0ce3bbfe"}
Sep 10 00:10:16.315769 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.315674    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d5f17c6ac1e1109fc026bd12cca9da01f01e9720f5fcbda44eaf5dc0ce3bbfe"
Sep 10 00:10:16.428212 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:16.428187    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"]
Sep 10 00:10:16.429422 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:10:16.429400    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95bf0a43_fe6d_4412_8eeb_ecc40c4a0391.slice/crio-4f83256380eef72a49a1f9b3f75e0bca6491c0c33451497f0c80bccc949c3dbe WatchSource:0}: Error finding container 4f83256380eef72a49a1f9b3f75e0bca6491c0c33451497f0c80bccc949c3dbe: Status 404 returned error can't find the container with id 4f83256380eef72a49a1f9b3f75e0bca6491c0c33451497f0c80bccc949c3dbe
Sep 10 00:10:17.322767 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:17.322729    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf" event={"ID":"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391","Type":"ContainerStarted","Data":"67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4"}
Sep 10 00:10:17.323188 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:17.322781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf" event={"ID":"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391","Type":"ContainerStarted","Data":"4f83256380eef72a49a1f9b3f75e0bca6491c0c33451497f0c80bccc949c3dbe"}
Sep 10 00:10:17.342314 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:17.342267    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf" podStartSLOduration=2.342249415 podStartE2EDuration="2.342249415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:10:17.341488692 +0000 UTC m=+1297.592624738" watchObservedRunningTime="2026-09-10 00:10:17.342249415 +0000 UTC m=+1297.593385458"
Sep 10 00:10:23.432848 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.432811    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"]
Sep 10 00:10:23.436992 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.436962    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:10:23.439484 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.439453    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-870d44-dockercfg-xnbst\""
Sep 10 00:10:23.439580 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.439463    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-870d44\""
Sep 10 00:10:23.447693 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.447665    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"]
Sep 10 00:10:23.451306 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.451284    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"]
Sep 10 00:10:23.455262 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.455246    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:10:23.467110 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.467089    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"]
Sep 10 00:10:23.494036 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.494007    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l46z\" (UniqueName: \"kubernetes.io/projected/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-kube-api-access-2l46z\") pod \"payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph\" (UID: \"1c9957ab-417f-4b86-8da4-d9b3480ff1ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:10:23.494163 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.494072    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1263ec79-e424-41dd-abaf-63445d26f97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2\" (UID: \"1263ec79-e424-41dd-abaf-63445d26f97a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:10:23.494163 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.494102    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph\" (UID: \"1c9957ab-417f-4b86-8da4-d9b3480ff1ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:10:23.494163 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.494145    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hblf\" (UniqueName: \"kubernetes.io/projected/1263ec79-e424-41dd-abaf-63445d26f97a-kube-api-access-6hblf\") pod \"payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2\" (UID: \"1263ec79-e424-41dd-abaf-63445d26f97a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:10:23.595538 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.595503    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hblf\" (UniqueName: \"kubernetes.io/projected/1263ec79-e424-41dd-abaf-63445d26f97a-kube-api-access-6hblf\") pod \"payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2\" (UID: \"1263ec79-e424-41dd-abaf-63445d26f97a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:10:23.595742 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.595632    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l46z\" (UniqueName: \"kubernetes.io/projected/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-kube-api-access-2l46z\") pod \"payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph\" (UID: \"1c9957ab-417f-4b86-8da4-d9b3480ff1ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:10:23.595742 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.595676    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1263ec79-e424-41dd-abaf-63445d26f97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2\" (UID: \"1263ec79-e424-41dd-abaf-63445d26f97a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:10:23.595742 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.595705    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph\" (UID: \"1c9957ab-417f-4b86-8da4-d9b3480ff1ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:10:23.596443 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.596422    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph\" (UID: \"1c9957ab-417f-4b86-8da4-d9b3480ff1ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:10:23.596518 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.596474    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1263ec79-e424-41dd-abaf-63445d26f97a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2\" (UID: \"1263ec79-e424-41dd-abaf-63445d26f97a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:10:23.604281 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.604255    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l46z\" (UniqueName: \"kubernetes.io/projected/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-kube-api-access-2l46z\") pod \"payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph\" (UID: \"1c9957ab-417f-4b86-8da4-d9b3480ff1ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:10:23.604419 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.604324    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hblf\" (UniqueName: \"kubernetes.io/projected/1263ec79-e424-41dd-abaf-63445d26f97a-kube-api-access-6hblf\") pod \"payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2\" (UID: \"1263ec79-e424-41dd-abaf-63445d26f97a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:10:23.646855 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.646818    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"]
Sep 10 00:10:23.647266 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.647097    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66" podUID="72e0c580-2c3b-4227-b870-870eada28626" containerName="payload-pre-processing" containerID="cri-o://d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e" gracePeriod=30
Sep 10 00:10:23.748804 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.748771    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:10:23.767392 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.767362    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:10:23.790842 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.790806    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"]
Sep 10 00:10:23.794893 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.794468    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f0f965-x2fsh"]
Sep 10 00:10:23.925867 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:23.925837    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:10:24.001119 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.001001    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwg9t\" (UniqueName: \"kubernetes.io/projected/72e0c580-2c3b-4227-b870-870eada28626-kube-api-access-pwg9t\") pod \"72e0c580-2c3b-4227-b870-870eada28626\" (UID: \"72e0c580-2c3b-4227-b870-870eada28626\") "
Sep 10 00:10:24.001119 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.001095    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e0c580-2c3b-4227-b870-870eada28626-plugins-config-volume\") pod \"72e0c580-2c3b-4227-b870-870eada28626\" (UID: \"72e0c580-2c3b-4227-b870-870eada28626\") "
Sep 10 00:10:24.001463 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.001435    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e0c580-2c3b-4227-b870-870eada28626-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72e0c580-2c3b-4227-b870-870eada28626" (UID: "72e0c580-2c3b-4227-b870-870eada28626"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:10:24.003154 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.003133    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e0c580-2c3b-4227-b870-870eada28626-kube-api-access-pwg9t" (OuterVolumeSpecName: "kube-api-access-pwg9t") pod "72e0c580-2c3b-4227-b870-870eada28626" (UID: "72e0c580-2c3b-4227-b870-870eada28626"). InnerVolumeSpecName "kube-api-access-pwg9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:10:24.036826 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.036799    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"]
Sep 10 00:10:24.037693 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:10:24.037666    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1263ec79_e424_41dd_abaf_63445d26f97a.slice/crio-1a6fb5b2b82fa477f4a9eb2eb33bae8830461d473bd46c3a8c5ce89cea2eb7bd WatchSource:0}: Error finding container 1a6fb5b2b82fa477f4a9eb2eb33bae8830461d473bd46c3a8c5ce89cea2eb7bd: Status 404 returned error can't find the container with id 1a6fb5b2b82fa477f4a9eb2eb33bae8830461d473bd46c3a8c5ce89cea2eb7bd
Sep 10 00:10:24.059592 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.059567    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"]
Sep 10 00:10:24.060154 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:10:24.060123    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9957ab_417f_4b86_8da4_d9b3480ff1ed.slice/crio-ccc110f488dc14502ee874aad9679d68e1b04dbbc8381f43c8183e776120c3d3 WatchSource:0}: Error finding container ccc110f488dc14502ee874aad9679d68e1b04dbbc8381f43c8183e776120c3d3: Status 404 returned error can't find the container with id ccc110f488dc14502ee874aad9679d68e1b04dbbc8381f43c8183e776120c3d3
Sep 10 00:10:24.102337 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.102314    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwg9t\" (UniqueName: \"kubernetes.io/projected/72e0c580-2c3b-4227-b870-870eada28626-kube-api-access-pwg9t\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:24.102337 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.102338    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72e0c580-2c3b-4227-b870-870eada28626-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:24.271977 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.271897    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef2ae9a9-41e4-450e-9b49-9297b3ed7afa" path="/var/lib/kubelet/pods/ef2ae9a9-41e4-450e-9b49-9297b3ed7afa/volumes"
Sep 10 00:10:24.367726 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.367687    2556 generic.go:356] "Generic (PLEG): container finished" podID="72e0c580-2c3b-4227-b870-870eada28626" containerID="d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e" exitCode=0
Sep 10 00:10:24.367895 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.367740    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"
Sep 10 00:10:24.367895 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.367771    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66" event={"ID":"72e0c580-2c3b-4227-b870-870eada28626","Type":"ContainerDied","Data":"d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e"}
Sep 10 00:10:24.367895 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.367849    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66" event={"ID":"72e0c580-2c3b-4227-b870-870eada28626","Type":"ContainerDied","Data":"ff4bbd81b468837291bd8bbf818595a5bebcc889ff5115fa1ccf2b407ff30ecb"}
Sep 10 00:10:24.367895 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.367875    2556 scope.go:117] "RemoveContainer" containerID="d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e"
Sep 10 00:10:24.369746 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.369716    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2" event={"ID":"1263ec79-e424-41dd-abaf-63445d26f97a","Type":"ContainerStarted","Data":"38bd695d373ed25390525763a37eb7b0d2a43f1b8ce5aad1642e7f473fdbfd0b"}
Sep 10 00:10:24.370611 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.369778    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:10:24.370611 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.369813    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2" event={"ID":"1263ec79-e424-41dd-abaf-63445d26f97a","Type":"ContainerStarted","Data":"1a6fb5b2b82fa477f4a9eb2eb33bae8830461d473bd46c3a8c5ce89cea2eb7bd"}
Sep 10 00:10:24.372034 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.372010    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph" event={"ID":"1c9957ab-417f-4b86-8da4-d9b3480ff1ed","Type":"ContainerStarted","Data":"786cfa1414ce85a309f17b8e8be7a95c47e4407cf307458bb798d76a64a898c4"}
Sep 10 00:10:24.372170 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.372082    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph" event={"ID":"1c9957ab-417f-4b86-8da4-d9b3480ff1ed","Type":"ContainerStarted","Data":"ccc110f488dc14502ee874aad9679d68e1b04dbbc8381f43c8183e776120c3d3"}
Sep 10 00:10:24.372170 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.372106    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:10:24.378460 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.378428    2556 scope.go:117] "RemoveContainer" containerID="d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e"
Sep 10 00:10:24.378736 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:10:24.378714    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e\": container with ID starting with d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e not found: ID does not exist" containerID="d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e"
Sep 10 00:10:24.378789 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.378743    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e"} err="failed to get container status \"d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e\": rpc error: code = NotFound desc = could not find container \"d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e\": container with ID starting with d121514705325890f195fbea5f4dd2ab5674128ead7d2c5c1343ad0e4bd1902e not found: ID does not exist"
Sep 10 00:10:24.385103 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.385077    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"]
Sep 10 00:10:24.387683 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.387662    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0f965-7bc76dc7c4-8nz66"]
Sep 10 00:10:24.401856 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.401821    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2" podStartSLOduration=1.401808535 podStartE2EDuration="1.401808535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:10:24.401524956 +0000 UTC m=+1304.652661001" watchObservedRunningTime="2026-09-10 00:10:24.401808535 +0000 UTC m=+1304.652944598"
Sep 10 00:10:24.422858 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:24.422813    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph" podStartSLOduration=1.42279968 podStartE2EDuration="1.42279968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:10:24.420572601 +0000 UTC m=+1304.671708647" watchObservedRunningTime="2026-09-10 00:10:24.42279968 +0000 UTC m=+1304.673935729"
Sep 10 00:10:26.156437 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:26.156401    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:26.156788 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:26.156456    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:26.169215 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:26.169187    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:26.272216 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:26.272189    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72e0c580-2c3b-4227-b870-870eada28626" path="/var/lib/kubelet/pods/72e0c580-2c3b-4227-b870-870eada28626/volumes"
Sep 10 00:10:26.396998 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:26.396963    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:10:30.495277 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.495242    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"]
Sep 10 00:10:30.495726 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.495711    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72e0c580-2c3b-4227-b870-870eada28626" containerName="payload-pre-processing"
Sep 10 00:10:30.495776 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.495727    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="72e0c580-2c3b-4227-b870-870eada28626" containerName="payload-pre-processing"
Sep 10 00:10:30.495820 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.495812    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="72e0c580-2c3b-4227-b870-870eada28626" containerName="payload-pre-processing"
Sep 10 00:10:30.500155 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.500136    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"
Sep 10 00:10:30.503260 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.503233    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:10:30.503468 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.503289    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:10:30.506870 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.506847    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"]
Sep 10 00:10:30.568881 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.568851    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77597d1e-030a-497c-a8f5-32f58569d4a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx\" (UID: \"77597d1e-030a-497c-a8f5-32f58569d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"
Sep 10 00:10:30.670017 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.669978    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77597d1e-030a-497c-a8f5-32f58569d4a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx\" (UID: \"77597d1e-030a-497c-a8f5-32f58569d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"
Sep 10 00:10:30.670613 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.670593    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77597d1e-030a-497c-a8f5-32f58569d4a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx\" (UID: \"77597d1e-030a-497c-a8f5-32f58569d4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"
Sep 10 00:10:30.740675 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.740613    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw" podUID="3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" containerName="main" containerID="cri-o://c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38" gracePeriod=45
Sep 10 00:10:30.814468 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:30.814436    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"
Sep 10 00:10:31.024829 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.024799    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:10:31.085898 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.085867    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"]
Sep 10 00:10:31.087335 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:10:31.087311    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77597d1e_030a_497c_a8f5_32f58569d4a5.slice/crio-8677fafd00d6aef7dced051f231c7ee171454b68ed321e83d9aa980ad6ff6040 WatchSource:0}: Error finding container 8677fafd00d6aef7dced051f231c7ee171454b68ed321e83d9aa980ad6ff6040: Status 404 returned error can't find the container with id 8677fafd00d6aef7dced051f231c7ee171454b68ed321e83d9aa980ad6ff6040
Sep 10 00:10:31.176840 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.176802    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tmp-dir\") pod \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") "
Sep 10 00:10:31.177034 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.176853    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tls-certs\") pod \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") "
Sep 10 00:10:31.177034 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.176906    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-447k9\" (UniqueName: \"kubernetes.io/projected/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-kube-api-access-447k9\") pod \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") "
Sep 10 00:10:31.177034 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.176991    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-dshm\") pod \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") "
Sep 10 00:10:31.177034 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.177012    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-model-cache\") pod \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") "
Sep 10 00:10:31.177286 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.177109    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-home\") pod \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\" (UID: \"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5\") "
Sep 10 00:10:31.177286 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.177110    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" (UID: "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:31.177431 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.177378    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" (UID: "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:31.177431 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.177425    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-home" (OuterVolumeSpecName: "home") pod "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" (UID: "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:31.177618 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.177469    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:31.179282 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.179259    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" (UID: "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:10:31.179282 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.179269    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-dshm" (OuterVolumeSpecName: "dshm") pod "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" (UID: "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:10:31.179413 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.179309    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-kube-api-access-447k9" (OuterVolumeSpecName: "kube-api-access-447k9") pod "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" (UID: "3bf1aff8-3204-4b86-a2cc-3440ec86d9f5"). InnerVolumeSpecName "kube-api-access-447k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:10:31.278777 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.278748    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:31.278777 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.278773    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-447k9\" (UniqueName: \"kubernetes.io/projected/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-kube-api-access-447k9\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:31.278969 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.278782    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:31.278969 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.278792    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:31.278969 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.278802    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:31.416579 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.416544    2556 generic.go:356] "Generic (PLEG): container finished" podID="3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" containerID="c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38" exitCode=0
Sep 10 00:10:31.416749 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.416585    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"
Sep 10 00:10:31.416749 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.416622    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw" event={"ID":"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5","Type":"ContainerDied","Data":"c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38"}
Sep 10 00:10:31.416749 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.416665    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw" event={"ID":"3bf1aff8-3204-4b86-a2cc-3440ec86d9f5","Type":"ContainerDied","Data":"e32f3fb06e3625034cf1c31c64b32aeffa7ba95ccbf732745c8143321185770c"}
Sep 10 00:10:31.416749 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.416682    2556 scope.go:117] "RemoveContainer" containerID="c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38"
Sep 10 00:10:31.418555 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.418526    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx" event={"ID":"77597d1e-030a-497c-a8f5-32f58569d4a5","Type":"ContainerStarted","Data":"5fe11325f5f4a12596c7e012016d8dfd242ba7927d06619f5de67442a19865a9"}
Sep 10 00:10:31.418674 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.418572    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx" event={"ID":"77597d1e-030a-497c-a8f5-32f58569d4a5","Type":"ContainerStarted","Data":"8677fafd00d6aef7dced051f231c7ee171454b68ed321e83d9aa980ad6ff6040"}
Sep 10 00:10:31.427619 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.427540    2556 scope.go:117] "RemoveContainer" containerID="c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38"
Sep 10 00:10:31.428808 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:10:31.428742    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38\": container with ID starting with c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38 not found: ID does not exist" containerID="c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38"
Sep 10 00:10:31.428808 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.428781    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38"} err="failed to get container status \"c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38\": rpc error: code = NotFound desc = could not find container \"c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38\": container with ID starting with c5079a82b1e134feb8845d8bea54f8c3cf5b3ab7f79257cdba05b9e95a7bdd38 not found: ID does not exist"
Sep 10 00:10:31.435642 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.435592    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx" podStartSLOduration=1.435576977 podStartE2EDuration="1.435576977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:10:31.433132611 +0000 UTC m=+1311.684268655" watchObservedRunningTime="2026-09-10 00:10:31.435576977 +0000 UTC m=+1311.686713021"
Sep 10 00:10:31.447503 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.447476    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"]
Sep 10 00:10:31.450915 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:31.450892    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c59d42/noten-model-c59d42-kserve-687bbfbf6c-58gfw"]
Sep 10 00:10:32.270357 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:32.270320    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" path="/var/lib/kubelet/pods/3bf1aff8-3204-4b86-a2cc-3440ec86d9f5/volumes"
Sep 10 00:10:32.426938 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:32.426891    2556 generic.go:356] "Generic (PLEG): container finished" podID="77597d1e-030a-497c-a8f5-32f58569d4a5" containerID="5fe11325f5f4a12596c7e012016d8dfd242ba7927d06619f5de67442a19865a9" exitCode=0
Sep 10 00:10:32.426938 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:32.426930    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx" event={"ID":"77597d1e-030a-497c-a8f5-32f58569d4a5","Type":"ContainerDied","Data":"5fe11325f5f4a12596c7e012016d8dfd242ba7927d06619f5de67442a19865a9"}
Sep 10 00:10:33.567264 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:33.567231    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"
Sep 10 00:10:33.706454 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:33.706427    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77597d1e-030a-497c-a8f5-32f58569d4a5-maas-api-service-ca\") pod \"77597d1e-030a-497c-a8f5-32f58569d4a5\" (UID: \"77597d1e-030a-497c-a8f5-32f58569d4a5\") "
Sep 10 00:10:33.706802 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:33.706778    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77597d1e-030a-497c-a8f5-32f58569d4a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "77597d1e-030a-497c-a8f5-32f58569d4a5" (UID: "77597d1e-030a-497c-a8f5-32f58569d4a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:10:33.807527 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:33.807495    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77597d1e-030a-497c-a8f5-32f58569d4a5-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:34.439583 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:34.439553    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"
Sep 10 00:10:34.439741 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:34.439592    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx" event={"ID":"77597d1e-030a-497c-a8f5-32f58569d4a5","Type":"ContainerDied","Data":"8677fafd00d6aef7dced051f231c7ee171454b68ed321e83d9aa980ad6ff6040"}
Sep 10 00:10:34.439741 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:34.439646    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8677fafd00d6aef7dced051f231c7ee171454b68ed321e83d9aa980ad6ff6040"
Sep 10 00:10:35.381807 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:35.381778    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:10:35.382291 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:35.381849    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:10:40.975006 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:40.974640    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"]
Sep 10 00:10:40.981556 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:40.981525    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"]
Sep 10 00:10:40.982003 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:40.981971    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56" podUID="27c2e4e5-8e38-4a87-b681-3aede504f838" containerName="payload-pre-processing" containerID="cri-o://031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5" gracePeriod=30
Sep 10 00:10:40.982952 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:40.982922    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp" podUID="3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43" containerName="payload-processing" containerID="cri-o://167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b" gracePeriod=30
Sep 10 00:10:41.138420 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.138388    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"]
Sep 10 00:10:41.141714 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.141685    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a6d968-tjlbx"]
Sep 10 00:10:41.268782 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.268750    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:10:41.271813 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.271792    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:10:41.372918 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.372889    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c2e4e5-8e38-4a87-b681-3aede504f838-plugins-config-volume\") pod \"27c2e4e5-8e38-4a87-b681-3aede504f838\" (UID: \"27c2e4e5-8e38-4a87-b681-3aede504f838\") "
Sep 10 00:10:41.373100 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.372931    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7psv\" (UniqueName: \"kubernetes.io/projected/27c2e4e5-8e38-4a87-b681-3aede504f838-kube-api-access-l7psv\") pod \"27c2e4e5-8e38-4a87-b681-3aede504f838\" (UID: \"27c2e4e5-8e38-4a87-b681-3aede504f838\") "
Sep 10 00:10:41.373100 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.373028    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-plugins-config-volume\") pod \"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43\" (UID: \"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43\") "
Sep 10 00:10:41.373247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.373118    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd2h6\" (UniqueName: \"kubernetes.io/projected/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-kube-api-access-zd2h6\") pod \"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43\" (UID: \"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43\") "
Sep 10 00:10:41.373354 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.373326    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c2e4e5-8e38-4a87-b681-3aede504f838-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27c2e4e5-8e38-4a87-b681-3aede504f838" (UID: "27c2e4e5-8e38-4a87-b681-3aede504f838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:10:41.373444 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.373421    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c2e4e5-8e38-4a87-b681-3aede504f838-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:41.373570 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.373430    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43" (UID: "3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:10:41.375205 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.375186    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c2e4e5-8e38-4a87-b681-3aede504f838-kube-api-access-l7psv" (OuterVolumeSpecName: "kube-api-access-l7psv") pod "27c2e4e5-8e38-4a87-b681-3aede504f838" (UID: "27c2e4e5-8e38-4a87-b681-3aede504f838"). InnerVolumeSpecName "kube-api-access-l7psv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:10:41.375205 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.375196    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-kube-api-access-zd2h6" (OuterVolumeSpecName: "kube-api-access-zd2h6") pod "3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43" (UID: "3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43"). InnerVolumeSpecName "kube-api-access-zd2h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:10:41.474540 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.474509    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:41.474628 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.474541    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd2h6\" (UniqueName: \"kubernetes.io/projected/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43-kube-api-access-zd2h6\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:41.474628 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.474557    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7psv\" (UniqueName: \"kubernetes.io/projected/27c2e4e5-8e38-4a87-b681-3aede504f838-kube-api-access-l7psv\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:41.480597 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.480573    2556 generic.go:356] "Generic (PLEG): container finished" podID="27c2e4e5-8e38-4a87-b681-3aede504f838" containerID="031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5" exitCode=0
Sep 10 00:10:41.480701 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.480618    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"
Sep 10 00:10:41.480701 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.480668    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56" event={"ID":"27c2e4e5-8e38-4a87-b681-3aede504f838","Type":"ContainerDied","Data":"031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5"}
Sep 10 00:10:41.480800 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.480724    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56" event={"ID":"27c2e4e5-8e38-4a87-b681-3aede504f838","Type":"ContainerDied","Data":"1f831c1de71497a1c7cb11ce2f1e521876981b7698a26cfa1f0bbf481e2f77f0"}
Sep 10 00:10:41.480800 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.480753    2556 scope.go:117] "RemoveContainer" containerID="031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5"
Sep 10 00:10:41.482189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.482000    2556 generic.go:356] "Generic (PLEG): container finished" podID="3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43" containerID="167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b" exitCode=0
Sep 10 00:10:41.482189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.482038    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"
Sep 10 00:10:41.482189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.482092    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp" event={"ID":"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43","Type":"ContainerDied","Data":"167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b"}
Sep 10 00:10:41.482189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.482143    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp" event={"ID":"3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43","Type":"ContainerDied","Data":"242f1bd8d8f4bb164203697bfbadc4444aa99224d13b3fcf2a7f794b94d4920f"}
Sep 10 00:10:41.491788 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.491769    2556 scope.go:117] "RemoveContainer" containerID="031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5"
Sep 10 00:10:41.492099 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:10:41.492077    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5\": container with ID starting with 031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5 not found: ID does not exist" containerID="031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5"
Sep 10 00:10:41.492189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.492105    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5"} err="failed to get container status \"031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5\": rpc error: code = NotFound desc = could not find container \"031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5\": container with ID starting with 031b0f003a3332c82c65f0e95f876d9d7d0d4713242289af54d5b488fcba2bf5 not found: ID does not exist"
Sep 10 00:10:41.492189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.492121    2556 scope.go:117] "RemoveContainer" containerID="167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b"
Sep 10 00:10:41.500372 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.500353    2556 scope.go:117] "RemoveContainer" containerID="167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b"
Sep 10 00:10:41.500646 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:10:41.500623    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b\": container with ID starting with 167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b not found: ID does not exist" containerID="167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b"
Sep 10 00:10:41.500704 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.500653    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b"} err="failed to get container status \"167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b\": rpc error: code = NotFound desc = could not find container \"167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b\": container with ID starting with 167981402f77bd3a71e94b0b404fd673d45ed8b735b3783573bef03f216fb69b not found: ID does not exist"
Sep 10 00:10:41.516795 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.516774    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"]
Sep 10 00:10:41.522771 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.522751    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a6d968-577f489cdb-h6g56"]
Sep 10 00:10:41.543976 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.543956    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"]
Sep 10 00:10:41.551678 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:41.551659    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a6d968-bb6cddf55-sg8tp"]
Sep 10 00:10:42.268136 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:42.268104    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c2e4e5-8e38-4a87-b681-3aede504f838" path="/var/lib/kubelet/pods/27c2e4e5-8e38-4a87-b681-3aede504f838/volumes"
Sep 10 00:10:42.268499 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:42.268461    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43" path="/var/lib/kubelet/pods/3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43/volumes"
Sep 10 00:10:42.268752 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:42.268739    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77597d1e-030a-497c-a8f5-32f58569d4a5" path="/var/lib/kubelet/pods/77597d1e-030a-497c-a8f5-32f58569d4a5/volumes"
Sep 10 00:10:51.666929 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.666894    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-e9a20b"]
Sep 10 00:10:51.667392 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667376    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43" containerName="payload-processing"
Sep 10 00:10:51.667392 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667393    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43" containerName="payload-processing"
Sep 10 00:10:51.667472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667403    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c2e4e5-8e38-4a87-b681-3aede504f838" containerName="payload-pre-processing"
Sep 10 00:10:51.667472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667409    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c2e4e5-8e38-4a87-b681-3aede504f838" containerName="payload-pre-processing"
Sep 10 00:10:51.667472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667423    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" containerName="main"
Sep 10 00:10:51.667472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667429    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" containerName="main"
Sep 10 00:10:51.667472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667445    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77597d1e-030a-497c-a8f5-32f58569d4a5" containerName="revoke-keys"
Sep 10 00:10:51.667472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667451    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="77597d1e-030a-497c-a8f5-32f58569d4a5" containerName="revoke-keys"
Sep 10 00:10:51.667659 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667514    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf1aff8-3204-4b86-a2cc-3440ec86d9f5" containerName="main"
Sep 10 00:10:51.667659 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667521    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="77597d1e-030a-497c-a8f5-32f58569d4a5" containerName="revoke-keys"
Sep 10 00:10:51.667659 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667530    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c2e4e5-8e38-4a87-b681-3aede504f838" containerName="payload-pre-processing"
Sep 10 00:10:51.667659 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.667537    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d9ad54e-34e0-4f57-9a5b-dbecd3de6e43" containerName="payload-processing"
Sep 10 00:10:51.672614 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.672595    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-e9a20b"
Sep 10 00:10:51.674916 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.674879    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gtwx5\""
Sep 10 00:10:51.676528 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.676504    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-e9a20b"]
Sep 10 00:10:51.764364 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.764340    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-768hc\" (UniqueName: \"kubernetes.io/projected/bd836ff2-b683-4441-8240-084a543ad738-kube-api-access-768hc\") pod \"test-curl-7789-e9a20b\" (UID: \"bd836ff2-b683-4441-8240-084a543ad738\") " pod="opendatahub/test-curl-7789-e9a20b"
Sep 10 00:10:51.865495 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.865459    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-768hc\" (UniqueName: \"kubernetes.io/projected/bd836ff2-b683-4441-8240-084a543ad738-kube-api-access-768hc\") pod \"test-curl-7789-e9a20b\" (UID: \"bd836ff2-b683-4441-8240-084a543ad738\") " pod="opendatahub/test-curl-7789-e9a20b"
Sep 10 00:10:51.873435 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.873409    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-768hc\" (UniqueName: \"kubernetes.io/projected/bd836ff2-b683-4441-8240-084a543ad738-kube-api-access-768hc\") pod \"test-curl-7789-e9a20b\" (UID: \"bd836ff2-b683-4441-8240-084a543ad738\") " pod="opendatahub/test-curl-7789-e9a20b"
Sep 10 00:10:51.986226 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:51.986201    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-e9a20b"
Sep 10 00:10:52.255617 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:52.255590    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-e9a20b"]
Sep 10 00:10:52.257257 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:10:52.257227    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd836ff2_b683_4441_8240_084a543ad738.slice/crio-069cf75332f82a4a849670f23654d6cc23417b35cd0624fc5d43d4c4398f4e77 WatchSource:0}: Error finding container 069cf75332f82a4a849670f23654d6cc23417b35cd0624fc5d43d4c4398f4e77: Status 404 returned error can't find the container with id 069cf75332f82a4a849670f23654d6cc23417b35cd0624fc5d43d4c4398f4e77
Sep 10 00:10:52.548300 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:52.548218    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-e9a20b" event={"ID":"bd836ff2-b683-4441-8240-084a543ad738","Type":"ContainerStarted","Data":"f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799"}
Sep 10 00:10:52.548300 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:52.548259    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-e9a20b" event={"ID":"bd836ff2-b683-4441-8240-084a543ad738","Type":"ContainerStarted","Data":"069cf75332f82a4a849670f23654d6cc23417b35cd0624fc5d43d4c4398f4e77"}
Sep 10 00:10:52.564250 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:52.564204    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7789-e9a20b" podStartSLOduration=1.564189942 podStartE2EDuration="1.564189942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:10:52.561675111 +0000 UTC m=+1332.812811155" watchObservedRunningTime="2026-09-10 00:10:52.564189942 +0000 UTC m=+1332.815325984"
Sep 10 00:10:53.146806 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.146774    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-e9a20b"]
Sep 10 00:10:53.151319 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.151292    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-e9a20b"]
Sep 10 00:10:53.279765 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.279732    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7789-e70ae2"]
Sep 10 00:10:53.280252 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.280233    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd836ff2-b683-4441-8240-084a543ad738" containerName="test-curl-7789-e9a20b"
Sep 10 00:10:53.280357 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.280253    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd836ff2-b683-4441-8240-084a543ad738" containerName="test-curl-7789-e9a20b"
Sep 10 00:10:53.280421 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.280399    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd836ff2-b683-4441-8240-084a543ad738" containerName="test-curl-7789-e9a20b"
Sep 10 00:10:53.283633 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.283615    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-e70ae2"
Sep 10 00:10:53.290208 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.290185    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-e70ae2"]
Sep 10 00:10:53.378100 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.378046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvrf\" (UniqueName: \"kubernetes.io/projected/78138d6d-981b-4a3c-8f47-78643ffc1985-kube-api-access-fpvrf\") pod \"test-curl-7789-e70ae2\" (UID: \"78138d6d-981b-4a3c-8f47-78643ffc1985\") " pod="opendatahub/test-curl-7789-e70ae2"
Sep 10 00:10:53.479249 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.479217    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvrf\" (UniqueName: \"kubernetes.io/projected/78138d6d-981b-4a3c-8f47-78643ffc1985-kube-api-access-fpvrf\") pod \"test-curl-7789-e70ae2\" (UID: \"78138d6d-981b-4a3c-8f47-78643ffc1985\") " pod="opendatahub/test-curl-7789-e70ae2"
Sep 10 00:10:53.490359 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.490329    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvrf\" (UniqueName: \"kubernetes.io/projected/78138d6d-981b-4a3c-8f47-78643ffc1985-kube-api-access-fpvrf\") pod \"test-curl-7789-e70ae2\" (UID: \"78138d6d-981b-4a3c-8f47-78643ffc1985\") " pod="opendatahub/test-curl-7789-e70ae2"
Sep 10 00:10:53.553750 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:10:53.553684    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7789-e9a20b" container="test-curl-7789-e9a20b"
Sep 10 00:10:53.595472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.595449    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-e70ae2"
Sep 10 00:10:53.886619 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:53.886593    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7789-e70ae2"]
Sep 10 00:10:53.888245 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:10:53.888214    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78138d6d_981b_4a3c_8f47_78643ffc1985.slice/crio-189a00341b1e08a76838724ce6b2cf5c27a8171733c002d0b8007ded1044ff8c WatchSource:0}: Error finding container 189a00341b1e08a76838724ce6b2cf5c27a8171733c002d0b8007ded1044ff8c: Status 404 returned error can't find the container with id 189a00341b1e08a76838724ce6b2cf5c27a8171733c002d0b8007ded1044ff8c
Sep 10 00:10:54.560315 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:54.560283    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-e70ae2" event={"ID":"78138d6d-981b-4a3c-8f47-78643ffc1985","Type":"ContainerStarted","Data":"4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1"}
Sep 10 00:10:54.560760 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:54.560325    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7789-e70ae2" event={"ID":"78138d6d-981b-4a3c-8f47-78643ffc1985","Type":"ContainerStarted","Data":"189a00341b1e08a76838724ce6b2cf5c27a8171733c002d0b8007ded1044ff8c"}
Sep 10 00:10:54.560760 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:54.560441    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7789-e9a20b" podUID="bd836ff2-b683-4441-8240-084a543ad738" containerName="test-curl-7789-e9a20b" containerID="cri-o://f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799" gracePeriod=2
Sep 10 00:10:54.574995 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:54.574950    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7789-e70ae2" podStartSLOduration=1.574937391 podStartE2EDuration="1.574937391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:10:54.57256504 +0000 UTC m=+1334.823701083" watchObservedRunningTime="2026-09-10 00:10:54.574937391 +0000 UTC m=+1334.826073434"
Sep 10 00:10:55.158068 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:55.158028    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7789-e70ae2"]
Sep 10 00:10:55.161817 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:55.161794    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7789-e70ae2"]
Sep 10 00:10:55.565782 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:10:55.565753    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7789-e70ae2" container="test-curl-7789-e70ae2"
Sep 10 00:10:56.572362 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:56.572324    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7789-e70ae2" podUID="78138d6d-981b-4a3c-8f47-78643ffc1985" containerName="test-curl-7789-e70ae2" containerID="cri-o://4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1" gracePeriod=2
Sep 10 00:10:56.732154 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:56.732129    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-e9a20b"
Sep 10 00:10:56.813413 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:56.813378    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-768hc\" (UniqueName: \"kubernetes.io/projected/bd836ff2-b683-4441-8240-084a543ad738-kube-api-access-768hc\") pod \"bd836ff2-b683-4441-8240-084a543ad738\" (UID: \"bd836ff2-b683-4441-8240-084a543ad738\") "
Sep 10 00:10:56.815516 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:56.815455    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd836ff2-b683-4441-8240-084a543ad738-kube-api-access-768hc" (OuterVolumeSpecName: "kube-api-access-768hc") pod "bd836ff2-b683-4441-8240-084a543ad738" (UID: "bd836ff2-b683-4441-8240-084a543ad738"). InnerVolumeSpecName "kube-api-access-768hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:10:56.914860 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:56.914831    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-768hc\" (UniqueName: \"kubernetes.io/projected/bd836ff2-b683-4441-8240-084a543ad738-kube-api-access-768hc\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:57.579009 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:57.578975    2556 generic.go:356] "Generic (PLEG): container finished" podID="bd836ff2-b683-4441-8240-084a543ad738" containerID="f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799" exitCode=137
Sep 10 00:10:57.579444 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:57.579017    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-e9a20b"
Sep 10 00:10:57.579444 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:57.579107    2556 scope.go:117] "RemoveContainer" containerID="f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799"
Sep 10 00:10:57.588897 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:57.588877    2556 scope.go:117] "RemoveContainer" containerID="f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799"
Sep 10 00:10:57.589213 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:10:57.589196    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799\": container with ID starting with f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799 not found: ID does not exist" containerID="f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799"
Sep 10 00:10:57.589271 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:57.589221    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799"} err="failed to get container status \"f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799\": rpc error: code = NotFound desc = could not find container \"f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799\": container with ID starting with f7e3683168a7183b5e31e2af90bbca6f30fc9289613a406118ecf065896d1799 not found: ID does not exist"
Sep 10 00:10:57.589860 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:10:57.589842    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7789-e9a20b\" is forbidden: User \"system:node:ip-10-0-139-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-211.ec2.internal' and this object" podUID="bd836ff2-b683-4441-8240-084a543ad738" pod="opendatahub/test-curl-7789-e9a20b"
Sep 10 00:10:58.268472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:58.268439    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd836ff2-b683-4441-8240-084a543ad738" path="/var/lib/kubelet/pods/bd836ff2-b683-4441-8240-084a543ad738/volumes"
Sep 10 00:10:58.754495 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:58.754474    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-e70ae2"
Sep 10 00:10:58.835860 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:58.835803    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpvrf\" (UniqueName: \"kubernetes.io/projected/78138d6d-981b-4a3c-8f47-78643ffc1985-kube-api-access-fpvrf\") pod \"78138d6d-981b-4a3c-8f47-78643ffc1985\" (UID: \"78138d6d-981b-4a3c-8f47-78643ffc1985\") "
Sep 10 00:10:58.837967 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:58.837942    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78138d6d-981b-4a3c-8f47-78643ffc1985-kube-api-access-fpvrf" (OuterVolumeSpecName: "kube-api-access-fpvrf") pod "78138d6d-981b-4a3c-8f47-78643ffc1985" (UID: "78138d6d-981b-4a3c-8f47-78643ffc1985"). InnerVolumeSpecName "kube-api-access-fpvrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:10:58.937360 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:58.937332    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpvrf\" (UniqueName: \"kubernetes.io/projected/78138d6d-981b-4a3c-8f47-78643ffc1985-kube-api-access-fpvrf\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:10:59.592038 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:59.592007    2556 generic.go:356] "Generic (PLEG): container finished" podID="78138d6d-981b-4a3c-8f47-78643ffc1985" containerID="4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1" exitCode=137
Sep 10 00:10:59.592038 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:59.592046    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7789-e70ae2"
Sep 10 00:10:59.592273 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:59.592101    2556 scope.go:117] "RemoveContainer" containerID="4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1"
Sep 10 00:10:59.605525 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:59.602021    2556 scope.go:117] "RemoveContainer" containerID="4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1"
Sep 10 00:10:59.605772 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:10:59.605625    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1\": container with ID starting with 4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1 not found: ID does not exist" containerID="4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1"
Sep 10 00:10:59.605842 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:10:59.605794    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1"} err="failed to get container status \"4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1\": rpc error: code = NotFound desc = could not find container \"4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1\": container with ID starting with 4af40c0220acbb913ba994726c265208807d44fed656f2c8dc4aca9e0bedc1e1 not found: ID does not exist"
Sep 10 00:10:59.609354 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:10:59.609326    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7789-e70ae2\" is forbidden: User \"system:node:ip-10-0-139-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-211.ec2.internal' and this object" podUID="78138d6d-981b-4a3c-8f47-78643ffc1985" pod="opendatahub/test-curl-7789-e70ae2"
Sep 10 00:11:00.267466 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:11:00.267422    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7789-e70ae2\" is forbidden: User \"system:node:ip-10-0-139-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-211.ec2.internal' and this object" podUID="78138d6d-981b-4a3c-8f47-78643ffc1985" pod="opendatahub/test-curl-7789-e70ae2"
Sep 10 00:11:00.269049 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.269016    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78138d6d-981b-4a3c-8f47-78643ffc1985" path="/var/lib/kubelet/pods/78138d6d-981b-4a3c-8f47-78643ffc1985/volumes"
Sep 10 00:11:00.658399 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.658320    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"]
Sep 10 00:11:00.658776 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.658763    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78138d6d-981b-4a3c-8f47-78643ffc1985" containerName="test-curl-7789-e70ae2"
Sep 10 00:11:00.658817 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.658778    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="78138d6d-981b-4a3c-8f47-78643ffc1985" containerName="test-curl-7789-e70ae2"
Sep 10 00:11:00.658861 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.658852    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="78138d6d-981b-4a3c-8f47-78643ffc1985" containerName="test-curl-7789-e70ae2"
Sep 10 00:11:00.663297 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.663280    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:11:00.666343 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.666323    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-557790f7-dockercfg-dcsbg\""
Sep 10 00:11:00.666465 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.666336    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-557790f7\""
Sep 10 00:11:00.673840 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.673816    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"]
Sep 10 00:11:00.753338 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.753302    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tff9m\" (UniqueName: \"kubernetes.io/projected/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-kube-api-access-tff9m\") pod \"payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8\" (UID: \"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:11:00.753507 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.753354    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8\" (UID: \"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:11:00.855497 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.854764    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tff9m\" (UniqueName: \"kubernetes.io/projected/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-kube-api-access-tff9m\") pod \"payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8\" (UID: \"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:11:00.855497 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.854826    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8\" (UID: \"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:11:00.855869 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.855845    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8\" (UID: \"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:11:00.863173 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.863143    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tff9m\" (UniqueName: \"kubernetes.io/projected/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-kube-api-access-tff9m\") pod \"payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8\" (UID: \"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:11:00.975040 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:00.975009    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:11:01.243992 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:01.243964    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"]
Sep 10 00:11:01.246128 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:11:01.246100    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa69dec8_bb54_49a6_8ce5_3f77fa4500cd.slice/crio-65e60d06209516402b2a19b6f0bfcb0158bc71b07bf3f2171f524969720e7c7a WatchSource:0}: Error finding container 65e60d06209516402b2a19b6f0bfcb0158bc71b07bf3f2171f524969720e7c7a: Status 404 returned error can't find the container with id 65e60d06209516402b2a19b6f0bfcb0158bc71b07bf3f2171f524969720e7c7a
Sep 10 00:11:01.607446 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:01.607362    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8" event={"ID":"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd","Type":"ContainerStarted","Data":"7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c"}
Sep 10 00:11:01.607446 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:01.607419    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8" event={"ID":"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd","Type":"ContainerStarted","Data":"65e60d06209516402b2a19b6f0bfcb0158bc71b07bf3f2171f524969720e7c7a"}
Sep 10 00:11:01.607446 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:01.607445    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:11:01.622818 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:01.622764    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8" podStartSLOduration=1.622750476 podStartE2EDuration="1.622750476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:11:01.621162653 +0000 UTC m=+1341.872298728" watchObservedRunningTime="2026-09-10 00:11:01.622750476 +0000 UTC m=+1341.873886549"
Sep 10 00:11:12.615672 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:12.615569    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:11:21.849759 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:21.849723    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"]
Sep 10 00:11:21.853715 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:21.853699    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:11:21.856182 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:21.856161    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-b1baac\""
Sep 10 00:11:21.863495 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:21.863475    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"]
Sep 10 00:11:21.955442 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:21.955412    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18a10ba-2fbc-4aad-98a8-1423938ee772-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l\" (UID: \"f18a10ba-2fbc-4aad-98a8-1423938ee772\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:11:21.955570 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:21.955453    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjfl\" (UniqueName: \"kubernetes.io/projected/f18a10ba-2fbc-4aad-98a8-1423938ee772-kube-api-access-pmjfl\") pod \"payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l\" (UID: \"f18a10ba-2fbc-4aad-98a8-1423938ee772\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:11:22.057072 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:22.057009    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18a10ba-2fbc-4aad-98a8-1423938ee772-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l\" (UID: \"f18a10ba-2fbc-4aad-98a8-1423938ee772\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:11:22.057072 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:22.057077    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjfl\" (UniqueName: \"kubernetes.io/projected/f18a10ba-2fbc-4aad-98a8-1423938ee772-kube-api-access-pmjfl\") pod \"payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l\" (UID: \"f18a10ba-2fbc-4aad-98a8-1423938ee772\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:11:22.057730 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:22.057704    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18a10ba-2fbc-4aad-98a8-1423938ee772-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l\" (UID: \"f18a10ba-2fbc-4aad-98a8-1423938ee772\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:11:22.064978 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:22.064945    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjfl\" (UniqueName: \"kubernetes.io/projected/f18a10ba-2fbc-4aad-98a8-1423938ee772-kube-api-access-pmjfl\") pod \"payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l\" (UID: \"f18a10ba-2fbc-4aad-98a8-1423938ee772\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:11:22.165606 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:22.165547    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:11:22.444245 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:22.444220    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"]
Sep 10 00:11:22.445731 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:11:22.445708    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18a10ba_2fbc_4aad_98a8_1423938ee772.slice/crio-b7e79f9d5bc958f9608a1e5bc45bf5157db61ce9b54e0e64724eaa490ebe876c WatchSource:0}: Error finding container b7e79f9d5bc958f9608a1e5bc45bf5157db61ce9b54e0e64724eaa490ebe876c: Status 404 returned error can't find the container with id b7e79f9d5bc958f9608a1e5bc45bf5157db61ce9b54e0e64724eaa490ebe876c
Sep 10 00:11:22.727113 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:22.727078    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l" event={"ID":"f18a10ba-2fbc-4aad-98a8-1423938ee772","Type":"ContainerStarted","Data":"410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4"}
Sep 10 00:11:22.727282 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:22.727137    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l" event={"ID":"f18a10ba-2fbc-4aad-98a8-1423938ee772","Type":"ContainerStarted","Data":"b7e79f9d5bc958f9608a1e5bc45bf5157db61ce9b54e0e64724eaa490ebe876c"}
Sep 10 00:11:22.727361 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:22.727285    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:11:22.744490 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:22.744444    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l" podStartSLOduration=1.744431938 podStartE2EDuration="1.744431938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:11:22.741703769 +0000 UTC m=+1362.992839824" watchObservedRunningTime="2026-09-10 00:11:22.744431938 +0000 UTC m=+1362.995567981"
Sep 10 00:11:33.736212 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:33.736176    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:11:53.457205 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.457165    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"]
Sep 10 00:11:53.462598 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.462578    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.465201 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.465176    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"unconfigured-w0-01b7d5-kserve-self-signed-certs\""
Sep 10 00:11:53.468888 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.468865    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"]
Sep 10 00:11:53.531350 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.531317    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvrlr\" (UniqueName: \"kubernetes.io/projected/d9225853-6085-4389-88b7-2bca95a12e4a-kube-api-access-vvrlr\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.531499 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.531393    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-dshm\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.531499 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.531441    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-model-cache\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.531499 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.531484    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9225853-6085-4389-88b7-2bca95a12e4a-tls-certs\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.531631 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.531510    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-home\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.531631 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.531536    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-tmp-dir\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.632407 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.632373    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-dshm\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.632551 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.632411    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-model-cache\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.632551 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.632440    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9225853-6085-4389-88b7-2bca95a12e4a-tls-certs\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.632551 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.632466    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-home\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.632551 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.632490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-tmp-dir\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.632747 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.632583    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvrlr\" (UniqueName: \"kubernetes.io/projected/d9225853-6085-4389-88b7-2bca95a12e4a-kube-api-access-vvrlr\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.632846 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.632826    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-model-cache\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.632915 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.632878    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-home\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.632980 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.632940    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-tmp-dir\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.634562 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.634542    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-dshm\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.634815 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.634798    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9225853-6085-4389-88b7-2bca95a12e4a-tls-certs\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.641318 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.641298    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvrlr\" (UniqueName: \"kubernetes.io/projected/d9225853-6085-4389-88b7-2bca95a12e4a-kube-api-access-vvrlr\") pod \"unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") " pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:53.773254 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:53.773196    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:11:54.059564 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:54.059532    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"]
Sep 10 00:11:54.060079 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:11:54.060036    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9225853_6085_4389_88b7_2bca95a12e4a.slice/crio-27499dc2694304610b10272ad917794ffac2ee22fb187a6f05b8f295d63a16e4 WatchSource:0}: Error finding container 27499dc2694304610b10272ad917794ffac2ee22fb187a6f05b8f295d63a16e4: Status 404 returned error can't find the container with id 27499dc2694304610b10272ad917794ffac2ee22fb187a6f05b8f295d63a16e4
Sep 10 00:11:54.920702 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:54.920669    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh" event={"ID":"d9225853-6085-4389-88b7-2bca95a12e4a","Type":"ContainerStarted","Data":"5f6ce2161791af0a97a3e817770a9e5e2868eaa6ea3d4035018314fd8adfb430"}
Sep 10 00:11:54.921116 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:54.921070    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh" event={"ID":"d9225853-6085-4389-88b7-2bca95a12e4a","Type":"ContainerStarted","Data":"27499dc2694304610b10272ad917794ffac2ee22fb187a6f05b8f295d63a16e4"}
Sep 10 00:11:54.941644 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:11:54.941597    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh" podStartSLOduration=1.941583343 podStartE2EDuration="1.941583343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:11:54.940779686 +0000 UTC m=+1395.191915721" watchObservedRunningTime="2026-09-10 00:11:54.941583343 +0000 UTC m=+1395.192719386"
Sep 10 00:12:03.773579 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:03.773537    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:12:03.773989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:03.773598    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:12:03.786087 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:03.786026    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:12:03.981445 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:03.981417    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:12:22.042505 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.042469    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"]
Sep 10 00:12:22.046446 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.046427    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.049180 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.049157    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc0040\"/\"unconfigured-w4-cc0040-kserve-self-signed-certs\""
Sep 10 00:12:22.054299 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.054180    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"]
Sep 10 00:12:22.097106 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.097046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdg6n\" (UniqueName: \"kubernetes.io/projected/35ac38ec-a740-4763-98b3-cbed9245cdfa-kube-api-access-hdg6n\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.097283 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.097121    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-dshm\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.097283 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.097147    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-home\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.097283 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.097190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-model-cache\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.097283 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.097246    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-tmp-dir\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.097447 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.097304    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ac38ec-a740-4763-98b3-cbed9245cdfa-tls-certs\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.198886 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.198852    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-model-cache\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.198886 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.198888    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-tmp-dir\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.199162 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.198942    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ac38ec-a740-4763-98b3-cbed9245cdfa-tls-certs\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.199162 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.199003    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdg6n\" (UniqueName: \"kubernetes.io/projected/35ac38ec-a740-4763-98b3-cbed9245cdfa-kube-api-access-hdg6n\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.199162 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.199038    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-dshm\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.199162 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.199107    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-home\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.199391 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.199315    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-model-cache\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.199391 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.199373    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-tmp-dir\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.199477 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.199453    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-home\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.201379 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.201353    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-dshm\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.201544 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.201526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ac38ec-a740-4763-98b3-cbed9245cdfa-tls-certs\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.206755 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.206734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdg6n\" (UniqueName: \"kubernetes.io/projected/35ac38ec-a740-4763-98b3-cbed9245cdfa-kube-api-access-hdg6n\") pod \"unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") " pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.358441 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.358382    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:22.631673 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:22.631646    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"]
Sep 10 00:12:22.634204 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:12:22.634175    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35ac38ec_a740_4763_98b3_cbed9245cdfa.slice/crio-82a619b78fd9074f040b678d657c0e093d1f1a18632c2ce6da4bb24ba8875825 WatchSource:0}: Error finding container 82a619b78fd9074f040b678d657c0e093d1f1a18632c2ce6da4bb24ba8875825: Status 404 returned error can't find the container with id 82a619b78fd9074f040b678d657c0e093d1f1a18632c2ce6da4bb24ba8875825
Sep 10 00:12:23.078424 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:23.078389    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh" event={"ID":"35ac38ec-a740-4763-98b3-cbed9245cdfa","Type":"ContainerStarted","Data":"cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e"}
Sep 10 00:12:23.078809 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:23.078431    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh" event={"ID":"35ac38ec-a740-4763-98b3-cbed9245cdfa","Type":"ContainerStarted","Data":"82a619b78fd9074f040b678d657c0e093d1f1a18632c2ce6da4bb24ba8875825"}
Sep 10 00:12:23.099282 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:23.099232    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh" podStartSLOduration=1.099219956 podStartE2EDuration="1.099219956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:12:23.097641805 +0000 UTC m=+1423.348777847" watchObservedRunningTime="2026-09-10 00:12:23.099219956 +0000 UTC m=+1423.350355999"
Sep 10 00:12:32.358880 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:32.358835    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:32.359409 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:32.358889    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:32.371307 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:32.371279    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:33.147880 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:33.147844    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:12:40.405983 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:12:40.405949    2556 scope.go:117] "RemoveContainer" containerID="4b5bdb629a3fddf01269bc3e106b0de0835c9439dafc0eba612b3a44d9268ddb"
Sep 10 00:13:35.955558 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:35.955522    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"]
Sep 10 00:13:35.960361 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:35.960328    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"
Sep 10 00:13:35.963240 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:35.963208    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:13:35.963970 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:35.963947    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:13:35.965770 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:35.965747    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"]
Sep 10 00:13:35.973205 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:35.971456    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:35.973205 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:35.971839    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"]
Sep 10 00:13:35.973940 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:35.973920    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-01b7d5\"/\"embedding-w0-01b7d5-kserve-self-signed-certs\""
Sep 10 00:13:35.982033 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:35.982011    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"]
Sep 10 00:13:36.075228 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.075195    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5bg\" (UniqueName: \"kubernetes.io/projected/166c136a-d51b-4ac0-87ca-eb685101acd3-kube-api-access-sb5bg\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.075395 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.075252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-model-cache\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.075395 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.075307    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/166c136a-d51b-4ac0-87ca-eb685101acd3-tls-certs\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.075395 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.075379    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-home\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.075620 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.075449    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-tmp-dir\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.075620 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.075502    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c2f052-9e52-4c06-a3e9-5553c344182e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-557790f7-6wzwj\" (UID: \"e0c2f052-9e52-4c06-a3e9-5553c344182e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"
Sep 10 00:13:36.075620 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.075606    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-dshm\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.176860 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.176834    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5bg\" (UniqueName: \"kubernetes.io/projected/166c136a-d51b-4ac0-87ca-eb685101acd3-kube-api-access-sb5bg\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.177012 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.176875    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-model-cache\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.177012 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.176892    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/166c136a-d51b-4ac0-87ca-eb685101acd3-tls-certs\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.177163 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.177011    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-home\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.177163 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.177122    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-tmp-dir\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.177163 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.177151    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c2f052-9e52-4c06-a3e9-5553c344182e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-557790f7-6wzwj\" (UID: \"e0c2f052-9e52-4c06-a3e9-5553c344182e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"
Sep 10 00:13:36.177298 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.177207    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-dshm\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.177348 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.177300    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-model-cache\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.177348 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.177338    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-home\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.177452 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.177429    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-tmp-dir\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.177760 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.177739    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c2f052-9e52-4c06-a3e9-5553c344182e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-557790f7-6wzwj\" (UID: \"e0c2f052-9e52-4c06-a3e9-5553c344182e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"
Sep 10 00:13:36.179286 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.179262    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-dshm\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.179406 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.179367    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/166c136a-d51b-4ac0-87ca-eb685101acd3-tls-certs\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.184818 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.184797    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5bg\" (UniqueName: \"kubernetes.io/projected/166c136a-d51b-4ac0-87ca-eb685101acd3-kube-api-access-sb5bg\") pod \"embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") " pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.277683 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.277620    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"
Sep 10 00:13:36.288770 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.288753    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:36.554432 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.554240    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"]
Sep 10 00:13:36.556829 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:13:36.556803    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c2f052_9e52_4c06_a3e9_5553c344182e.slice/crio-fa2bba5d5f8551bc7b304ed18c5fbb35a8cbe9e4ad1de745c26f20a91ad4ef9b WatchSource:0}: Error finding container fa2bba5d5f8551bc7b304ed18c5fbb35a8cbe9e4ad1de745c26f20a91ad4ef9b: Status 404 returned error can't find the container with id fa2bba5d5f8551bc7b304ed18c5fbb35a8cbe9e4ad1de745c26f20a91ad4ef9b
Sep 10 00:13:36.571989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:36.571967    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"]
Sep 10 00:13:36.573301 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:13:36.573276    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod166c136a_d51b_4ac0_87ca_eb685101acd3.slice/crio-71fa9d9844309561426a790c95c98cbc30c04ac80391bc761d475a6e759d826b WatchSource:0}: Error finding container 71fa9d9844309561426a790c95c98cbc30c04ac80391bc761d475a6e759d826b: Status 404 returned error can't find the container with id 71fa9d9844309561426a790c95c98cbc30c04ac80391bc761d475a6e759d826b
Sep 10 00:13:37.487913 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:37.487886    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6" event={"ID":"166c136a-d51b-4ac0-87ca-eb685101acd3","Type":"ContainerStarted","Data":"a0a24e0d5fa9a232bae253a2f4ef53f7d154da3db7c6a9aed6a602ccf7d8aa6f"}
Sep 10 00:13:37.488338 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:37.487928    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6" event={"ID":"166c136a-d51b-4ac0-87ca-eb685101acd3","Type":"ContainerStarted","Data":"71fa9d9844309561426a790c95c98cbc30c04ac80391bc761d475a6e759d826b"}
Sep 10 00:13:37.489627 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:37.489603    2556 generic.go:356] "Generic (PLEG): container finished" podID="e0c2f052-9e52-4c06-a3e9-5553c344182e" containerID="fc28416498bc641da0deb7e1d96fdb94b9226e4de4edefa5031eb39ab8b9121f" exitCode=0
Sep 10 00:13:37.489706 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:37.489636    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj" event={"ID":"e0c2f052-9e52-4c06-a3e9-5553c344182e","Type":"ContainerDied","Data":"fc28416498bc641da0deb7e1d96fdb94b9226e4de4edefa5031eb39ab8b9121f"}
Sep 10 00:13:37.489706 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:37.489667    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj" event={"ID":"e0c2f052-9e52-4c06-a3e9-5553c344182e","Type":"ContainerStarted","Data":"fa2bba5d5f8551bc7b304ed18c5fbb35a8cbe9e4ad1de745c26f20a91ad4ef9b"}
Sep 10 00:13:37.506525 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:37.506445    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6" podStartSLOduration=2.50643136 podStartE2EDuration="2.50643136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:13:37.504636403 +0000 UTC m=+1497.755772447" watchObservedRunningTime="2026-09-10 00:13:37.50643136 +0000 UTC m=+1497.757567443"
Sep 10 00:13:38.630488 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:38.630466    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"
Sep 10 00:13:38.798020 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:38.797958    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c2f052-9e52-4c06-a3e9-5553c344182e-maas-api-service-ca\") pod \"e0c2f052-9e52-4c06-a3e9-5553c344182e\" (UID: \"e0c2f052-9e52-4c06-a3e9-5553c344182e\") "
Sep 10 00:13:38.798335 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:38.798313    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c2f052-9e52-4c06-a3e9-5553c344182e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0c2f052-9e52-4c06-a3e9-5553c344182e" (UID: "e0c2f052-9e52-4c06-a3e9-5553c344182e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:13:38.899096 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:38.899049    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c2f052-9e52-4c06-a3e9-5553c344182e-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:13:39.504806 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:39.504780    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"
Sep 10 00:13:39.504974 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:39.504772    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj" event={"ID":"e0c2f052-9e52-4c06-a3e9-5553c344182e","Type":"ContainerDied","Data":"fa2bba5d5f8551bc7b304ed18c5fbb35a8cbe9e4ad1de745c26f20a91ad4ef9b"}
Sep 10 00:13:39.504974 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:39.504942    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa2bba5d5f8551bc7b304ed18c5fbb35a8cbe9e4ad1de745c26f20a91ad4ef9b"
Sep 10 00:13:46.289248 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:46.289214    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:46.289708 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:46.289260    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:46.301578 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:46.301555    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:46.554662 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:46.554578    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:13:48.053197 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.053159    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"]
Sep 10 00:13:48.053843 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.053820    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c2f052-9e52-4c06-a3e9-5553c344182e" containerName="revoke-keys"
Sep 10 00:13:48.053843 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.053844    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c2f052-9e52-4c06-a3e9-5553c344182e" containerName="revoke-keys"
Sep 10 00:13:48.054024 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.053984    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c2f052-9e52-4c06-a3e9-5553c344182e" containerName="revoke-keys"
Sep 10 00:13:48.057238 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.057217    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.059681 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.059657    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc0040\"/\"embedding-w4-cc0040-kserve-self-signed-certs\""
Sep 10 00:13:48.067116 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.067092    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"]
Sep 10 00:13:48.080401 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.080375    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-tmp-dir\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.080514 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.080432    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223834c5-5b10-471b-9e03-7bbd3aa817a9-tls-certs\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.080514 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.080491    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-dshm\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.080600 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.080580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-model-cache\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.080677 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.080657    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbw8w\" (UniqueName: \"kubernetes.io/projected/223834c5-5b10-471b-9e03-7bbd3aa817a9-kube-api-access-nbw8w\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.080748 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.080732    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-home\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.182135 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.182103    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-dshm\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.182247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.182162    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-model-cache\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.182247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.182200    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbw8w\" (UniqueName: \"kubernetes.io/projected/223834c5-5b10-471b-9e03-7bbd3aa817a9-kube-api-access-nbw8w\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.182247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.182240    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-home\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.182426 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.182276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-tmp-dir\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.182426 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.182304    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223834c5-5b10-471b-9e03-7bbd3aa817a9-tls-certs\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.182632 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.182610    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-model-cache\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.182700 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.182670    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-home\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.182777 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.182741    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-tmp-dir\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.184324 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.184301    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-dshm\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.184744 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.184727    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223834c5-5b10-471b-9e03-7bbd3aa817a9-tls-certs\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.190110 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.190092    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbw8w\" (UniqueName: \"kubernetes.io/projected/223834c5-5b10-471b-9e03-7bbd3aa817a9-kube-api-access-nbw8w\") pod \"embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") " pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.367845 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.367779    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:48.637856 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:48.637825    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"]
Sep 10 00:13:48.638988 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:13:48.638962    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod223834c5_5b10_471b_9e03_7bbd3aa817a9.slice/crio-d4fc43ec7790d0bf6478dd6d3f6eac899f370813f89932e5aa6e63d416ca6937 WatchSource:0}: Error finding container d4fc43ec7790d0bf6478dd6d3f6eac899f370813f89932e5aa6e63d416ca6937: Status 404 returned error can't find the container with id d4fc43ec7790d0bf6478dd6d3f6eac899f370813f89932e5aa6e63d416ca6937
Sep 10 00:13:49.561202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:49.561162    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn" event={"ID":"223834c5-5b10-471b-9e03-7bbd3aa817a9","Type":"ContainerStarted","Data":"9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161"}
Sep 10 00:13:49.561557 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:49.561213    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn" event={"ID":"223834c5-5b10-471b-9e03-7bbd3aa817a9","Type":"ContainerStarted","Data":"d4fc43ec7790d0bf6478dd6d3f6eac899f370813f89932e5aa6e63d416ca6937"}
Sep 10 00:13:49.580011 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:49.579965    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn" podStartSLOduration=1.579952119 podStartE2EDuration="1.579952119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:13:49.57892495 +0000 UTC m=+1509.830060997" watchObservedRunningTime="2026-09-10 00:13:49.579952119 +0000 UTC m=+1509.831088163"
Sep 10 00:13:51.541100 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:51.541062    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"]
Sep 10 00:13:51.541556 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:51.541297    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8" podUID="fa69dec8-bb54-49a6-8ce5-3f77fa4500cd" containerName="payload-pre-processing" containerID="cri-o://7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c" gracePeriod=30
Sep 10 00:13:51.724570 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:51.724499    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"]
Sep 10 00:13:51.726705 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:51.726680    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-557790f7-6wzwj"]
Sep 10 00:13:51.797459 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:51.797401    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:13:51.819798 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:51.819767    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-plugins-config-volume\") pod \"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd\" (UID: \"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd\") "
Sep 10 00:13:51.819925 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:51.819884    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tff9m\" (UniqueName: \"kubernetes.io/projected/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-kube-api-access-tff9m\") pod \"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd\" (UID: \"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd\") "
Sep 10 00:13:51.820212 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:51.820188    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa69dec8-bb54-49a6-8ce5-3f77fa4500cd" (UID: "fa69dec8-bb54-49a6-8ce5-3f77fa4500cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:13:51.820355 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:51.820290    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:13:51.822605 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:51.822243    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-kube-api-access-tff9m" (OuterVolumeSpecName: "kube-api-access-tff9m") pod "fa69dec8-bb54-49a6-8ce5-3f77fa4500cd" (UID: "fa69dec8-bb54-49a6-8ce5-3f77fa4500cd"). InnerVolumeSpecName "kube-api-access-tff9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:13:51.921147 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:51.921120    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tff9m\" (UniqueName: \"kubernetes.io/projected/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd-kube-api-access-tff9m\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:13:52.269261 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:52.269230    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c2f052-9e52-4c06-a3e9-5553c344182e" path="/var/lib/kubelet/pods/e0c2f052-9e52-4c06-a3e9-5553c344182e/volumes"
Sep 10 00:13:52.581155 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:52.581040    2556 generic.go:356] "Generic (PLEG): container finished" podID="fa69dec8-bb54-49a6-8ce5-3f77fa4500cd" containerID="7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c" exitCode=0
Sep 10 00:13:52.581155 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:52.581098    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8" event={"ID":"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd","Type":"ContainerDied","Data":"7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c"}
Sep 10 00:13:52.581155 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:52.581128    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"
Sep 10 00:13:52.581155 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:52.581157    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8" event={"ID":"fa69dec8-bb54-49a6-8ce5-3f77fa4500cd","Type":"ContainerDied","Data":"65e60d06209516402b2a19b6f0bfcb0158bc71b07bf3f2171f524969720e7c7a"}
Sep 10 00:13:52.581727 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:52.581181    2556 scope.go:117] "RemoveContainer" containerID="7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c"
Sep 10 00:13:52.591305 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:52.591288    2556 scope.go:117] "RemoveContainer" containerID="7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c"
Sep 10 00:13:52.591650 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:13:52.591626    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c\": container with ID starting with 7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c not found: ID does not exist" containerID="7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c"
Sep 10 00:13:52.591697 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:52.591659    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c"} err="failed to get container status \"7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c\": rpc error: code = NotFound desc = could not find container \"7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c\": container with ID starting with 7a843b9d375f712576966172726de96b059c95150c3ae4737745ebbbf2a4294c not found: ID does not exist"
Sep 10 00:13:52.601182 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:52.601160    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"]
Sep 10 00:13:52.610193 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:52.610168    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-557790f7-5d45b9884b-q4fj8"]
Sep 10 00:13:54.268924 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:54.268890    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa69dec8-bb54-49a6-8ce5-3f77fa4500cd" path="/var/lib/kubelet/pods/fa69dec8-bb54-49a6-8ce5-3f77fa4500cd/volumes"
Sep 10 00:13:58.368791 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:58.368760    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:58.369184 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:58.368806    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:58.381006 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:58.380984    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:13:58.629706 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:13:58.629631    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:15:00.144018 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.143987    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc"]
Sep 10 00:15:00.144505 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.144484    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa69dec8-bb54-49a6-8ce5-3f77fa4500cd" containerName="payload-pre-processing"
Sep 10 00:15:00.144505 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.144497    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa69dec8-bb54-49a6-8ce5-3f77fa4500cd" containerName="payload-pre-processing"
Sep 10 00:15:00.144587 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.144571    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa69dec8-bb54-49a6-8ce5-3f77fa4500cd" containerName="payload-pre-processing"
Sep 10 00:15:00.148188 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.148166    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl"]
Sep 10 00:15:00.148312 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.148292    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc"
Sep 10 00:15:00.150731 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.150709    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:15:00.151622 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.151608    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk"]
Sep 10 00:15:00.151756 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.151737    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl"
Sep 10 00:15:00.155261 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.154968    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq"]
Sep 10 00:15:00.155261 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.155047    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk"
Sep 10 00:15:00.159422 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.159401    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc"]
Sep 10 00:15:00.159574 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.159494    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq"
Sep 10 00:15:00.164520 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.164169    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc"
Sep 10 00:15:00.165424 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.165395    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl"]
Sep 10 00:15:00.170819 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.168828    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk"]
Sep 10 00:15:00.172323 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.172303    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq"]
Sep 10 00:15:00.176480 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.176461    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl"
Sep 10 00:15:00.177442 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.177422    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr"]
Sep 10 00:15:00.185625 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.185609    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr"]
Sep 10 00:15:00.187324 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.186219    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr"
Sep 10 00:15:00.211648 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.211600    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk"
Sep 10 00:15:00.223569 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.223395    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq"
Sep 10 00:15:00.277990 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.277942    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr"
Sep 10 00:15:00.458294 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.458268    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc"]
Sep 10 00:15:00.460681 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:15:00.460656    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cbb693_2ba8_4a19_b776_c657bd6e0ead.slice/crio-3937b9deda86389bdccce29865d7a28de2d32c9390c0fa5797e5fd15fc0b5df3 WatchSource:0}: Error finding container 3937b9deda86389bdccce29865d7a28de2d32c9390c0fa5797e5fd15fc0b5df3: Status 404 returned error can't find the container with id 3937b9deda86389bdccce29865d7a28de2d32c9390c0fa5797e5fd15fc0b5df3
Sep 10 00:15:00.470259 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.468501    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl"]
Sep 10 00:15:00.470546 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:15:00.470370    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad82862_c1ab_45b0_8c30_a65bd667d340.slice/crio-8679bc58daa76c3b2970c02b3466066153d5acaae71a89cd66f907a8cebeb180 WatchSource:0}: Error finding container 8679bc58daa76c3b2970c02b3466066153d5acaae71a89cd66f907a8cebeb180: Status 404 returned error can't find the container with id 8679bc58daa76c3b2970c02b3466066153d5acaae71a89cd66f907a8cebeb180
Sep 10 00:15:00.513519 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.513492    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk"]
Sep 10 00:15:00.515068 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:15:00.515022    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c52e85_fe8c_4539_a6db_e6e9e2448465.slice/crio-4da72dd6b13f195955dac0b96392a91f84ba687fbe917a0dc7139a7b974a7c6c WatchSource:0}: Error finding container 4da72dd6b13f195955dac0b96392a91f84ba687fbe917a0dc7139a7b974a7c6c: Status 404 returned error can't find the container with id 4da72dd6b13f195955dac0b96392a91f84ba687fbe917a0dc7139a7b974a7c6c
Sep 10 00:15:00.532128 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.532107    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq"]
Sep 10 00:15:00.533317 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:15:00.533294    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ba1b84_15ab_46c8_9b01_b790ae35cc90.slice/crio-977b410600497823964493de607f21cf67cff86ffce9ce824ee12b1da4a6c499 WatchSource:0}: Error finding container 977b410600497823964493de607f21cf67cff86ffce9ce824ee12b1da4a6c499: Status 404 returned error can't find the container with id 977b410600497823964493de607f21cf67cff86ffce9ce824ee12b1da4a6c499
Sep 10 00:15:00.564801 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.564782    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr"]
Sep 10 00:15:00.645669 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:15:00.645640    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa08f2f2_1184_449c_983f_d26a8921ab0d.slice/crio-88958640f1ce27006f98d291707c6d832bb98f61ba360fed13dc07c63da2f213 WatchSource:0}: Error finding container 88958640f1ce27006f98d291707c6d832bb98f61ba360fed13dc07c63da2f213: Status 404 returned error can't find the container with id 88958640f1ce27006f98d291707c6d832bb98f61ba360fed13dc07c63da2f213
Sep 10 00:15:00.995358 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.995326    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl" event={"ID":"fad82862-c1ab-45b0-8c30-a65bd667d340","Type":"ContainerStarted","Data":"8bf82cca5ee9d341f027512c12b28039fa3d04a34089f500ee922ace97d6e0a2"}
Sep 10 00:15:00.995506 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.995423    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl" event={"ID":"fad82862-c1ab-45b0-8c30-a65bd667d340","Type":"ContainerStarted","Data":"8679bc58daa76c3b2970c02b3466066153d5acaae71a89cd66f907a8cebeb180"}
Sep 10 00:15:00.996608 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.996582    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr" event={"ID":"aa08f2f2-1184-449c-983f-d26a8921ab0d","Type":"ContainerStarted","Data":"88958640f1ce27006f98d291707c6d832bb98f61ba360fed13dc07c63da2f213"}
Sep 10 00:15:00.998186 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.998158    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk" event={"ID":"c4c52e85-fe8c-4539-a6db-e6e9e2448465","Type":"ContainerStarted","Data":"688db57725beda236fb62fdcd897c9f6f9fe9e236aa6a5ebb5be6241451d99b1"}
Sep 10 00:15:00.998284 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.998211    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk" event={"ID":"c4c52e85-fe8c-4539-a6db-e6e9e2448465","Type":"ContainerStarted","Data":"4da72dd6b13f195955dac0b96392a91f84ba687fbe917a0dc7139a7b974a7c6c"}
Sep 10 00:15:00.999667 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.999647    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc" event={"ID":"b9cbb693-2ba8-4a19-b776-c657bd6e0ead","Type":"ContainerStarted","Data":"0730d3831444e9a00c6330bf30e2f2667db415ae33ac6e5ae711444e3c8493ac"}
Sep 10 00:15:00.999770 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:00.999688    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc" event={"ID":"b9cbb693-2ba8-4a19-b776-c657bd6e0ead","Type":"ContainerStarted","Data":"3937b9deda86389bdccce29865d7a28de2d32c9390c0fa5797e5fd15fc0b5df3"}
Sep 10 00:15:01.001094 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:01.001048    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq" event={"ID":"31ba1b84-15ab-46c8-9b01-b790ae35cc90","Type":"ContainerStarted","Data":"b0ff77ec7d1688f56b902a367dee5ee47b870f9a7d9166cf4985ca29e64e15d4"}
Sep 10 00:15:01.001177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:01.001116    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq" event={"ID":"31ba1b84-15ab-46c8-9b01-b790ae35cc90","Type":"ContainerStarted","Data":"977b410600497823964493de607f21cf67cff86ffce9ce824ee12b1da4a6c499"}
Sep 10 00:15:01.010473 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:01.010408    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl" podStartSLOduration=1.010394614 podStartE2EDuration="1.010394614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:15:01.009040782 +0000 UTC m=+1581.260176825" watchObservedRunningTime="2026-09-10 00:15:01.010394614 +0000 UTC m=+1581.261530706"
Sep 10 00:15:01.024347 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:01.024308    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc" podStartSLOduration=1.024295961 podStartE2EDuration="1.024295961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:15:01.022493479 +0000 UTC m=+1581.273629522" watchObservedRunningTime="2026-09-10 00:15:01.024295961 +0000 UTC m=+1581.275432003"
Sep 10 00:15:01.037822 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:01.037782    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk" podStartSLOduration=1.037771486 podStartE2EDuration="1.037771486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:15:01.036595076 +0000 UTC m=+1581.287731119" watchObservedRunningTime="2026-09-10 00:15:01.037771486 +0000 UTC m=+1581.288907529"
Sep 10 00:15:01.051984 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:01.051951    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq" podStartSLOduration=1.051940073 podStartE2EDuration="1.051940073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:15:01.050381909 +0000 UTC m=+1581.301517952" watchObservedRunningTime="2026-09-10 00:15:01.051940073 +0000 UTC m=+1581.303076115"
Sep 10 00:15:02.010799 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:02.010762    2556 generic.go:356] "Generic (PLEG): container finished" podID="b9cbb693-2ba8-4a19-b776-c657bd6e0ead" containerID="0730d3831444e9a00c6330bf30e2f2667db415ae33ac6e5ae711444e3c8493ac" exitCode=0
Sep 10 00:15:02.011279 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:02.010816    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc" event={"ID":"b9cbb693-2ba8-4a19-b776-c657bd6e0ead","Type":"ContainerDied","Data":"0730d3831444e9a00c6330bf30e2f2667db415ae33ac6e5ae711444e3c8493ac"}
Sep 10 00:15:02.012544 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:02.012510    2556 generic.go:356] "Generic (PLEG): container finished" podID="31ba1b84-15ab-46c8-9b01-b790ae35cc90" containerID="b0ff77ec7d1688f56b902a367dee5ee47b870f9a7d9166cf4985ca29e64e15d4" exitCode=0
Sep 10 00:15:02.012690 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:02.012593    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq" event={"ID":"31ba1b84-15ab-46c8-9b01-b790ae35cc90","Type":"ContainerDied","Data":"b0ff77ec7d1688f56b902a367dee5ee47b870f9a7d9166cf4985ca29e64e15d4"}
Sep 10 00:15:02.014219 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:02.014197    2556 generic.go:356] "Generic (PLEG): container finished" podID="fad82862-c1ab-45b0-8c30-a65bd667d340" containerID="8bf82cca5ee9d341f027512c12b28039fa3d04a34089f500ee922ace97d6e0a2" exitCode=0
Sep 10 00:15:02.014388 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:02.014271    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl" event={"ID":"fad82862-c1ab-45b0-8c30-a65bd667d340","Type":"ContainerDied","Data":"8bf82cca5ee9d341f027512c12b28039fa3d04a34089f500ee922ace97d6e0a2"}
Sep 10 00:15:02.015904 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:02.015880    2556 generic.go:356] "Generic (PLEG): container finished" podID="aa08f2f2-1184-449c-983f-d26a8921ab0d" containerID="7dc294a72b75c729302de837f8f589893b12eb6af8aba4aed7bf847945cd46a9" exitCode=0
Sep 10 00:15:02.016003 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:02.015949    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr" event={"ID":"aa08f2f2-1184-449c-983f-d26a8921ab0d","Type":"ContainerDied","Data":"7dc294a72b75c729302de837f8f589893b12eb6af8aba4aed7bf847945cd46a9"}
Sep 10 00:15:02.017479 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:02.017459    2556 generic.go:356] "Generic (PLEG): container finished" podID="c4c52e85-fe8c-4539-a6db-e6e9e2448465" containerID="688db57725beda236fb62fdcd897c9f6f9fe9e236aa6a5ebb5be6241451d99b1" exitCode=0
Sep 10 00:15:02.017592 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:02.017515    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk" event={"ID":"c4c52e85-fe8c-4539-a6db-e6e9e2448465","Type":"ContainerDied","Data":"688db57725beda236fb62fdcd897c9f6f9fe9e236aa6a5ebb5be6241451d99b1"}
Sep 10 00:15:03.197282 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:03.197258    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr"
Sep 10 00:15:03.236212 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:03.236186    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq"
Sep 10 00:15:03.293399 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:03.293372    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc"
Sep 10 00:15:03.296903 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:03.296879    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl"
Sep 10 00:15:03.300380 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:03.300359    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk"
Sep 10 00:15:04.030966 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.030933    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl"
Sep 10 00:15:04.030966 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.030965    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl" event={"ID":"fad82862-c1ab-45b0-8c30-a65bd667d340","Type":"ContainerDied","Data":"8679bc58daa76c3b2970c02b3466066153d5acaae71a89cd66f907a8cebeb180"}
Sep 10 00:15:04.031250 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.031020    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8679bc58daa76c3b2970c02b3466066153d5acaae71a89cd66f907a8cebeb180"
Sep 10 00:15:04.032337 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.032314    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr"
Sep 10 00:15:04.032454 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.032339    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr" event={"ID":"aa08f2f2-1184-449c-983f-d26a8921ab0d","Type":"ContainerDied","Data":"88958640f1ce27006f98d291707c6d832bb98f61ba360fed13dc07c63da2f213"}
Sep 10 00:15:04.032454 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.032373    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88958640f1ce27006f98d291707c6d832bb98f61ba360fed13dc07c63da2f213"
Sep 10 00:15:04.033812 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.033785    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk" event={"ID":"c4c52e85-fe8c-4539-a6db-e6e9e2448465","Type":"ContainerDied","Data":"4da72dd6b13f195955dac0b96392a91f84ba687fbe917a0dc7139a7b974a7c6c"}
Sep 10 00:15:04.033924 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.033838    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4da72dd6b13f195955dac0b96392a91f84ba687fbe917a0dc7139a7b974a7c6c"
Sep 10 00:15:04.033924 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.033798    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk"
Sep 10 00:15:04.035180 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.035163    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc"
Sep 10 00:15:04.035180 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.035170    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc" event={"ID":"b9cbb693-2ba8-4a19-b776-c657bd6e0ead","Type":"ContainerDied","Data":"3937b9deda86389bdccce29865d7a28de2d32c9390c0fa5797e5fd15fc0b5df3"}
Sep 10 00:15:04.035327 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.035207    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3937b9deda86389bdccce29865d7a28de2d32c9390c0fa5797e5fd15fc0b5df3"
Sep 10 00:15:04.036583 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.036564    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq" event={"ID":"31ba1b84-15ab-46c8-9b01-b790ae35cc90","Type":"ContainerDied","Data":"977b410600497823964493de607f21cf67cff86ffce9ce824ee12b1da4a6c499"}
Sep 10 00:15:04.036699 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.036584    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq"
Sep 10 00:15:04.036699 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:04.036596    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="977b410600497823964493de607f21cf67cff86ffce9ce824ee12b1da4a6c499"
Sep 10 00:15:40.451123 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:15:40.451092    2556 scope.go:117] "RemoveContainer" containerID="beac0204d9449f0e3bd36e468bb5ca90a1fd5b8cd535111d7deec40f1d2d3d25"
Sep 10 00:16:13.830923 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.830879    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"]
Sep 10 00:16:13.831711 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831687    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ba1b84-15ab-46c8-9b01-b790ae35cc90" containerName="cleanup"
Sep 10 00:16:13.831766 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831716    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ba1b84-15ab-46c8-9b01-b790ae35cc90" containerName="cleanup"
Sep 10 00:16:13.831766 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831752    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4c52e85-fe8c-4539-a6db-e6e9e2448465" containerName="cleanup"
Sep 10 00:16:13.831766 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831761    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4c52e85-fe8c-4539-a6db-e6e9e2448465" containerName="cleanup"
Sep 10 00:16:13.831860 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831779    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9cbb693-2ba8-4a19-b776-c657bd6e0ead" containerName="cleanup"
Sep 10 00:16:13.831860 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831788    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9cbb693-2ba8-4a19-b776-c657bd6e0ead" containerName="cleanup"
Sep 10 00:16:13.831860 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831808    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fad82862-c1ab-45b0-8c30-a65bd667d340" containerName="cleanup"
Sep 10 00:16:13.831860 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831816    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fad82862-c1ab-45b0-8c30-a65bd667d340" containerName="cleanup"
Sep 10 00:16:13.831860 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831826    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa08f2f2-1184-449c-983f-d26a8921ab0d" containerName="cleanup"
Sep 10 00:16:13.831860 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831835    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa08f2f2-1184-449c-983f-d26a8921ab0d" containerName="cleanup"
Sep 10 00:16:13.832040 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831941    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9cbb693-2ba8-4a19-b776-c657bd6e0ead" containerName="cleanup"
Sep 10 00:16:13.832040 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831954    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4c52e85-fe8c-4539-a6db-e6e9e2448465" containerName="cleanup"
Sep 10 00:16:13.832040 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831966    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ba1b84-15ab-46c8-9b01-b790ae35cc90" containerName="cleanup"
Sep 10 00:16:13.832040 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831980    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fad82862-c1ab-45b0-8c30-a65bd667d340" containerName="cleanup"
Sep 10 00:16:13.832040 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.831991    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa08f2f2-1184-449c-983f-d26a8921ab0d" containerName="cleanup"
Sep 10 00:16:13.835708 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.835683    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:16:13.838349 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.838328    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-beac5e-dockercfg-db6hg\""
Sep 10 00:16:13.838455 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.838329    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-beac5e\""
Sep 10 00:16:13.846694 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.846666    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"]
Sep 10 00:16:13.934305 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.934274    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq5gj\" (UniqueName: \"kubernetes.io/projected/f5475663-2301-40d0-9937-4202f3d35ba8-kube-api-access-xq5gj\") pod \"payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn\" (UID: \"f5475663-2301-40d0-9937-4202f3d35ba8\") " pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:16:13.934425 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:13.934390    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5475663-2301-40d0-9937-4202f3d35ba8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn\" (UID: \"f5475663-2301-40d0-9937-4202f3d35ba8\") " pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:16:14.035928 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:14.035896    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq5gj\" (UniqueName: \"kubernetes.io/projected/f5475663-2301-40d0-9937-4202f3d35ba8-kube-api-access-xq5gj\") pod \"payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn\" (UID: \"f5475663-2301-40d0-9937-4202f3d35ba8\") " pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:16:14.036103 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:14.035954    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5475663-2301-40d0-9937-4202f3d35ba8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn\" (UID: \"f5475663-2301-40d0-9937-4202f3d35ba8\") " pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:16:14.036639 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:14.036621    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5475663-2301-40d0-9937-4202f3d35ba8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn\" (UID: \"f5475663-2301-40d0-9937-4202f3d35ba8\") " pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:16:14.045949 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:14.045915    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq5gj\" (UniqueName: \"kubernetes.io/projected/f5475663-2301-40d0-9937-4202f3d35ba8-kube-api-access-xq5gj\") pod \"payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn\" (UID: \"f5475663-2301-40d0-9937-4202f3d35ba8\") " pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:16:14.148558 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:14.148486    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:16:14.423764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:14.423740    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"]
Sep 10 00:16:14.425495 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:16:14.425468    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5475663_2301_40d0_9937_4202f3d35ba8.slice/crio-ce915f933e7b3c5111558a904ff963566f252754f8a598924f92a8ed952e35bf WatchSource:0}: Error finding container ce915f933e7b3c5111558a904ff963566f252754f8a598924f92a8ed952e35bf: Status 404 returned error can't find the container with id ce915f933e7b3c5111558a904ff963566f252754f8a598924f92a8ed952e35bf
Sep 10 00:16:14.460636 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:14.460601    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn" event={"ID":"f5475663-2301-40d0-9937-4202f3d35ba8","Type":"ContainerStarted","Data":"ce915f933e7b3c5111558a904ff963566f252754f8a598924f92a8ed952e35bf"}
Sep 10 00:16:15.467825 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:15.467793    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn" event={"ID":"f5475663-2301-40d0-9937-4202f3d35ba8","Type":"ContainerStarted","Data":"3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd"}
Sep 10 00:16:15.468220 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:15.467926    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:16:15.484962 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:15.484908    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn" podStartSLOduration=2.484893362 podStartE2EDuration="2.484893362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:16:15.483022154 +0000 UTC m=+1655.734158202" watchObservedRunningTime="2026-09-10 00:16:15.484893362 +0000 UTC m=+1655.736029404"
Sep 10 00:16:26.475240 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:26.475211    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:16:40.472111 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:40.472079    2556 scope.go:117] "RemoveContainer" containerID="5fe11325f5f4a12596c7e012016d8dfd242ba7927d06619f5de67442a19865a9"
Sep 10 00:16:40.480916 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:16:40.480896    2556 scope.go:117] "RemoveContainer" containerID="0e9c41e902ee25abc828f10afdef3ae6eb72507435207dcb057f5ebe43eec4d0"
Sep 10 00:17:09.413984 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:09.413886    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"]
Sep 10 00:17:09.418258 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:09.418236    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"
Sep 10 00:17:09.420587 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:09.420566    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:17:09.420728 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:09.420566    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:17:09.427104 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:09.426777    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"]
Sep 10 00:17:09.566915 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:09.566882    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec86057-a2af-436d-84da-9198d6c1606d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-beac5e-59q4k\" (UID: \"3ec86057-a2af-436d-84da-9198d6c1606d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"
Sep 10 00:17:09.667612 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:09.667530    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec86057-a2af-436d-84da-9198d6c1606d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-beac5e-59q4k\" (UID: \"3ec86057-a2af-436d-84da-9198d6c1606d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"
Sep 10 00:17:09.668213 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:09.668189    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec86057-a2af-436d-84da-9198d6c1606d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-beac5e-59q4k\" (UID: \"3ec86057-a2af-436d-84da-9198d6c1606d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"
Sep 10 00:17:09.730444 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:09.730419    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"
Sep 10 00:17:10.003999 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:10.003972    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"]
Sep 10 00:17:10.005299 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:17:10.005270    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec86057_a2af_436d_84da_9198d6c1606d.slice/crio-e700505f6c6a9350fe8bb89a916ee9a1884c56f4ac059ff9fd624d8a7f2774a3 WatchSource:0}: Error finding container e700505f6c6a9350fe8bb89a916ee9a1884c56f4ac059ff9fd624d8a7f2774a3: Status 404 returned error can't find the container with id e700505f6c6a9350fe8bb89a916ee9a1884c56f4ac059ff9fd624d8a7f2774a3
Sep 10 00:17:10.798240 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:10.798213    2556 generic.go:356] "Generic (PLEG): container finished" podID="3ec86057-a2af-436d-84da-9198d6c1606d" containerID="bc34750a7f76d2d9cc202e89e7c14f5ab211f36b9aaddd44d0848a60e58ad3b2" exitCode=0
Sep 10 00:17:10.798548 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:10.798280    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k" event={"ID":"3ec86057-a2af-436d-84da-9198d6c1606d","Type":"ContainerDied","Data":"bc34750a7f76d2d9cc202e89e7c14f5ab211f36b9aaddd44d0848a60e58ad3b2"}
Sep 10 00:17:10.798548 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:10.798315    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k" event={"ID":"3ec86057-a2af-436d-84da-9198d6c1606d","Type":"ContainerStarted","Data":"e700505f6c6a9350fe8bb89a916ee9a1884c56f4ac059ff9fd624d8a7f2774a3"}
Sep 10 00:17:11.946038 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:11.946009    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"
Sep 10 00:17:11.990715 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:11.990689    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec86057-a2af-436d-84da-9198d6c1606d-maas-api-service-ca\") pod \"3ec86057-a2af-436d-84da-9198d6c1606d\" (UID: \"3ec86057-a2af-436d-84da-9198d6c1606d\") "
Sep 10 00:17:11.991012 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:11.990990    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec86057-a2af-436d-84da-9198d6c1606d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ec86057-a2af-436d-84da-9198d6c1606d" (UID: "3ec86057-a2af-436d-84da-9198d6c1606d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:17:12.091542 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:12.091463    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec86057-a2af-436d-84da-9198d6c1606d-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:17:12.813227 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:12.813192    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"
Sep 10 00:17:12.813395 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:12.813226    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k" event={"ID":"3ec86057-a2af-436d-84da-9198d6c1606d","Type":"ContainerDied","Data":"e700505f6c6a9350fe8bb89a916ee9a1884c56f4ac059ff9fd624d8a7f2774a3"}
Sep 10 00:17:12.813395 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:12.813284    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e700505f6c6a9350fe8bb89a916ee9a1884c56f4ac059ff9fd624d8a7f2774a3"
Sep 10 00:17:24.947147 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:24.947111    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"]
Sep 10 00:17:24.947629 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:24.947373    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn" podUID="f5475663-2301-40d0-9937-4202f3d35ba8" containerName="payload-processing" containerID="cri-o://3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd" gracePeriod=30
Sep 10 00:17:25.099241 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.099202    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"]
Sep 10 00:17:25.101807 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.101781    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-beac5e-59q4k"]
Sep 10 00:17:25.207957 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.207934    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:17:25.322544 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.322513    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xq5gj\" (UniqueName: \"kubernetes.io/projected/f5475663-2301-40d0-9937-4202f3d35ba8-kube-api-access-xq5gj\") pod \"f5475663-2301-40d0-9937-4202f3d35ba8\" (UID: \"f5475663-2301-40d0-9937-4202f3d35ba8\") "
Sep 10 00:17:25.322707 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.322570    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5475663-2301-40d0-9937-4202f3d35ba8-plugins-config-volume\") pod \"f5475663-2301-40d0-9937-4202f3d35ba8\" (UID: \"f5475663-2301-40d0-9937-4202f3d35ba8\") "
Sep 10 00:17:25.322930 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.322906    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5475663-2301-40d0-9937-4202f3d35ba8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5475663-2301-40d0-9937-4202f3d35ba8" (UID: "f5475663-2301-40d0-9937-4202f3d35ba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:17:25.324562 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.324543    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5475663-2301-40d0-9937-4202f3d35ba8-kube-api-access-xq5gj" (OuterVolumeSpecName: "kube-api-access-xq5gj") pod "f5475663-2301-40d0-9937-4202f3d35ba8" (UID: "f5475663-2301-40d0-9937-4202f3d35ba8"). InnerVolumeSpecName "kube-api-access-xq5gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:17:25.423933 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.423901    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq5gj\" (UniqueName: \"kubernetes.io/projected/f5475663-2301-40d0-9937-4202f3d35ba8-kube-api-access-xq5gj\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:17:25.423933 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.423927    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5475663-2301-40d0-9937-4202f3d35ba8-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:17:25.893838 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.893802    2556 generic.go:356] "Generic (PLEG): container finished" podID="f5475663-2301-40d0-9937-4202f3d35ba8" containerID="3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd" exitCode=0
Sep 10 00:17:25.894018 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.893854    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"
Sep 10 00:17:25.894018 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.893890    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn" event={"ID":"f5475663-2301-40d0-9937-4202f3d35ba8","Type":"ContainerDied","Data":"3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd"}
Sep 10 00:17:25.894018 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.893953    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn" event={"ID":"f5475663-2301-40d0-9937-4202f3d35ba8","Type":"ContainerDied","Data":"ce915f933e7b3c5111558a904ff963566f252754f8a598924f92a8ed952e35bf"}
Sep 10 00:17:25.894018 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.893977    2556 scope.go:117] "RemoveContainer" containerID="3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd"
Sep 10 00:17:25.903848 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.903832    2556 scope.go:117] "RemoveContainer" containerID="3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd"
Sep 10 00:17:25.904113 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:17:25.904092    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd\": container with ID starting with 3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd not found: ID does not exist" containerID="3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd"
Sep 10 00:17:25.904163 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.904121    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd"} err="failed to get container status \"3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd\": rpc error: code = NotFound desc = could not find container \"3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd\": container with ID starting with 3349dc1031a0487c6c4fe9fc53e355a312ae0c26500730ee0d9203017913f3cd not found: ID does not exist"
Sep 10 00:17:25.916211 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.916185    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"]
Sep 10 00:17:25.921466 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:25.921445    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-beac5e-f654464b5-kvlrn"]
Sep 10 00:17:26.269409 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:26.269378    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec86057-a2af-436d-84da-9198d6c1606d" path="/var/lib/kubelet/pods/3ec86057-a2af-436d-84da-9198d6c1606d/volumes"
Sep 10 00:17:26.269796 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:26.269781    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5475663-2301-40d0-9937-4202f3d35ba8" path="/var/lib/kubelet/pods/f5475663-2301-40d0-9937-4202f3d35ba8/volumes"
Sep 10 00:17:31.895869 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.895832    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"]
Sep 10 00:17:31.896426 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.896404    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5475663-2301-40d0-9937-4202f3d35ba8" containerName="payload-processing"
Sep 10 00:17:31.896426 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.896426    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5475663-2301-40d0-9937-4202f3d35ba8" containerName="payload-processing"
Sep 10 00:17:31.896559 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.896440    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec86057-a2af-436d-84da-9198d6c1606d" containerName="revoke-keys"
Sep 10 00:17:31.896559 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.896446    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec86057-a2af-436d-84da-9198d6c1606d" containerName="revoke-keys"
Sep 10 00:17:31.896559 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.896524    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec86057-a2af-436d-84da-9198d6c1606d" containerName="revoke-keys"
Sep 10 00:17:31.896559 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.896535    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5475663-2301-40d0-9937-4202f3d35ba8" containerName="payload-processing"
Sep 10 00:17:31.901164 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.901143    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"
Sep 10 00:17:31.903771 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.903745    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:17:31.903998 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.903980    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:17:31.906637 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.906615    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"]
Sep 10 00:17:31.991682 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:31.991647    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738da4bd-6c90-489a-bbc9-0fefdf6f9c35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd\" (UID: \"738da4bd-6c90-489a-bbc9-0fefdf6f9c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"
Sep 10 00:17:32.092467 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:32.092434    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738da4bd-6c90-489a-bbc9-0fefdf6f9c35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd\" (UID: \"738da4bd-6c90-489a-bbc9-0fefdf6f9c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"
Sep 10 00:17:32.093081 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:32.093046    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738da4bd-6c90-489a-bbc9-0fefdf6f9c35-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd\" (UID: \"738da4bd-6c90-489a-bbc9-0fefdf6f9c35\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"
Sep 10 00:17:32.213616 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:32.213586    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"
Sep 10 00:17:32.484476 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:17:32.484447    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738da4bd_6c90_489a_bbc9_0fefdf6f9c35.slice/crio-7903a5534be578c3680e73df932371fb28195f296cd1be06b87559b07c777e18 WatchSource:0}: Error finding container 7903a5534be578c3680e73df932371fb28195f296cd1be06b87559b07c777e18: Status 404 returned error can't find the container with id 7903a5534be578c3680e73df932371fb28195f296cd1be06b87559b07c777e18
Sep 10 00:17:32.484615 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:32.484483    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"]
Sep 10 00:17:32.938443 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:32.938405    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd" event={"ID":"738da4bd-6c90-489a-bbc9-0fefdf6f9c35","Type":"ContainerStarted","Data":"0b7cbfddd7946faac6149ed2a3c98d317c1e53ab3a12d2e34287d40977a9a94c"}
Sep 10 00:17:32.938862 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:32.938468    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd" event={"ID":"738da4bd-6c90-489a-bbc9-0fefdf6f9c35","Type":"ContainerStarted","Data":"7903a5534be578c3680e73df932371fb28195f296cd1be06b87559b07c777e18"}
Sep 10 00:17:32.954561 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:32.954507    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd" podStartSLOduration=1.954488912 podStartE2EDuration="1.954488912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:17:32.951293485 +0000 UTC m=+1733.202429527" watchObservedRunningTime="2026-09-10 00:17:32.954488912 +0000 UTC m=+1733.205624954"
Sep 10 00:17:33.945337 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:33.945297    2556 generic.go:356] "Generic (PLEG): container finished" podID="738da4bd-6c90-489a-bbc9-0fefdf6f9c35" containerID="0b7cbfddd7946faac6149ed2a3c98d317c1e53ab3a12d2e34287d40977a9a94c" exitCode=0
Sep 10 00:17:33.945695 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:33.945384    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd" event={"ID":"738da4bd-6c90-489a-bbc9-0fefdf6f9c35","Type":"ContainerDied","Data":"0b7cbfddd7946faac6149ed2a3c98d317c1e53ab3a12d2e34287d40977a9a94c"}
Sep 10 00:17:35.091303 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:35.091275    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"
Sep 10 00:17:35.226524 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:35.226437    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738da4bd-6c90-489a-bbc9-0fefdf6f9c35-maas-api-service-ca\") pod \"738da4bd-6c90-489a-bbc9-0fefdf6f9c35\" (UID: \"738da4bd-6c90-489a-bbc9-0fefdf6f9c35\") "
Sep 10 00:17:35.226760 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:35.226733    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738da4bd-6c90-489a-bbc9-0fefdf6f9c35-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "738da4bd-6c90-489a-bbc9-0fefdf6f9c35" (UID: "738da4bd-6c90-489a-bbc9-0fefdf6f9c35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:17:35.226886 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:35.226872    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/738da4bd-6c90-489a-bbc9-0fefdf6f9c35-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:17:35.960507 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:35.960477    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"
Sep 10 00:17:35.960689 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:35.960512    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd" event={"ID":"738da4bd-6c90-489a-bbc9-0fefdf6f9c35","Type":"ContainerDied","Data":"7903a5534be578c3680e73df932371fb28195f296cd1be06b87559b07c777e18"}
Sep 10 00:17:35.960689 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:35.960567    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7903a5534be578c3680e73df932371fb28195f296cd1be06b87559b07c777e18"
Sep 10 00:17:47.514720 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:47.514681    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"]
Sep 10 00:17:47.519590 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:47.519555    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9483e-8dqwd"]
Sep 10 00:17:48.277232 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:48.277197    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="738da4bd-6c90-489a-bbc9-0fefdf6f9c35" path="/var/lib/kubelet/pods/738da4bd-6c90-489a-bbc9-0fefdf6f9c35/volumes"
Sep 10 00:17:54.099440 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.099403    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"]
Sep 10 00:17:54.099882 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.099868    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="738da4bd-6c90-489a-bbc9-0fefdf6f9c35" containerName="revoke-keys"
Sep 10 00:17:54.099882 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.099883    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="738da4bd-6c90-489a-bbc9-0fefdf6f9c35" containerName="revoke-keys"
Sep 10 00:17:54.099974 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.099965    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="738da4bd-6c90-489a-bbc9-0fefdf6f9c35" containerName="revoke-keys"
Sep 10 00:17:54.103330 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.103315    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"
Sep 10 00:17:54.105851 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.105823    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:17:54.106096 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.105866    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:17:54.110644 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.110620    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"]
Sep 10 00:17:54.123760 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.123728    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbde7fa-5963-43c2-aba8-b1427aed3653-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq\" (UID: \"4fbde7fa-5963-43c2-aba8-b1427aed3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"
Sep 10 00:17:54.224992 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.224961    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbde7fa-5963-43c2-aba8-b1427aed3653-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq\" (UID: \"4fbde7fa-5963-43c2-aba8-b1427aed3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"
Sep 10 00:17:54.225570 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.225549    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbde7fa-5963-43c2-aba8-b1427aed3653-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq\" (UID: \"4fbde7fa-5963-43c2-aba8-b1427aed3653\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"
Sep 10 00:17:54.416155 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.416076    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"
Sep 10 00:17:54.688804 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:54.688772    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"]
Sep 10 00:17:54.690460 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:17:54.690433    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbde7fa_5963_43c2_aba8_b1427aed3653.slice/crio-3addc4ae260dd602f273bb4d621ab8b8b8fbdf973d028f17cd3249619956c673 WatchSource:0}: Error finding container 3addc4ae260dd602f273bb4d621ab8b8b8fbdf973d028f17cd3249619956c673: Status 404 returned error can't find the container with id 3addc4ae260dd602f273bb4d621ab8b8b8fbdf973d028f17cd3249619956c673
Sep 10 00:17:55.082511 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:55.082459    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq" event={"ID":"4fbde7fa-5963-43c2-aba8-b1427aed3653","Type":"ContainerStarted","Data":"36a53d22703886975aac629b50e5c8b0c20d990f3fcd10de140f7adc55ad8b6a"}
Sep 10 00:17:55.082716 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:55.082572    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq" event={"ID":"4fbde7fa-5963-43c2-aba8-b1427aed3653","Type":"ContainerStarted","Data":"3addc4ae260dd602f273bb4d621ab8b8b8fbdf973d028f17cd3249619956c673"}
Sep 10 00:17:55.097582 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:55.097506    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq" podStartSLOduration=1.097491476 podStartE2EDuration="1.097491476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:17:55.095758386 +0000 UTC m=+1755.346894432" watchObservedRunningTime="2026-09-10 00:17:55.097491476 +0000 UTC m=+1755.348627519"
Sep 10 00:17:56.092327 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:56.092289    2556 generic.go:356] "Generic (PLEG): container finished" podID="4fbde7fa-5963-43c2-aba8-b1427aed3653" containerID="36a53d22703886975aac629b50e5c8b0c20d990f3fcd10de140f7adc55ad8b6a" exitCode=0
Sep 10 00:17:56.092756 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:56.092375    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq" event={"ID":"4fbde7fa-5963-43c2-aba8-b1427aed3653","Type":"ContainerDied","Data":"36a53d22703886975aac629b50e5c8b0c20d990f3fcd10de140f7adc55ad8b6a"}
Sep 10 00:17:57.237142 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:57.237113    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"
Sep 10 00:17:57.356787 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:57.356698    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbde7fa-5963-43c2-aba8-b1427aed3653-maas-api-service-ca\") pod \"4fbde7fa-5963-43c2-aba8-b1427aed3653\" (UID: \"4fbde7fa-5963-43c2-aba8-b1427aed3653\") "
Sep 10 00:17:57.357099 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:57.357046    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbde7fa-5963-43c2-aba8-b1427aed3653-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4fbde7fa-5963-43c2-aba8-b1427aed3653" (UID: "4fbde7fa-5963-43c2-aba8-b1427aed3653"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:17:57.357641 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:57.357539    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbde7fa-5963-43c2-aba8-b1427aed3653-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:17:58.105082 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:58.105032    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq" event={"ID":"4fbde7fa-5963-43c2-aba8-b1427aed3653","Type":"ContainerDied","Data":"3addc4ae260dd602f273bb4d621ab8b8b8fbdf973d028f17cd3249619956c673"}
Sep 10 00:17:58.105082 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:58.105066    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"
Sep 10 00:17:58.105295 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:17:58.105097    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3addc4ae260dd602f273bb4d621ab8b8b8fbdf973d028f17cd3249619956c673"
Sep 10 00:18:04.529253 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:04.529215    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc"]
Sep 10 00:18:04.533231 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:04.533198    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-1d36d9-29816655-c6vzc"]
Sep 10 00:18:04.707820 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:04.707771    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"]
Sep 10 00:18:04.710189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:04.710092    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1d36d9-dnprq"]
Sep 10 00:18:06.269020 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:06.268985    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fbde7fa-5963-43c2-aba8-b1427aed3653" path="/var/lib/kubelet/pods/4fbde7fa-5963-43c2-aba8-b1427aed3653/volumes"
Sep 10 00:18:06.269408 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:06.269325    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9cbb693-2ba8-4a19-b776-c657bd6e0ead" path="/var/lib/kubelet/pods/b9cbb693-2ba8-4a19-b776-c657bd6e0ead/volumes"
Sep 10 00:18:11.539909 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.539871    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"]
Sep 10 00:18:11.540678 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.540655    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fbde7fa-5963-43c2-aba8-b1427aed3653" containerName="revoke-keys"
Sep 10 00:18:11.540728 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.540683    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fbde7fa-5963-43c2-aba8-b1427aed3653" containerName="revoke-keys"
Sep 10 00:18:11.540795 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.540782    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fbde7fa-5963-43c2-aba8-b1427aed3653" containerName="revoke-keys"
Sep 10 00:18:11.544947 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.544929    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"
Sep 10 00:18:11.547434 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.547230    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:18:11.547434 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.547236    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:18:11.549068 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.549021    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"]
Sep 10 00:18:11.699598 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.699561    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f402f7-0491-4d80-b0d6-a746821880ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4\" (UID: \"86f402f7-0491-4d80-b0d6-a746821880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"
Sep 10 00:18:11.800644 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.800547    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f402f7-0491-4d80-b0d6-a746821880ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4\" (UID: \"86f402f7-0491-4d80-b0d6-a746821880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"
Sep 10 00:18:11.801350 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.801321    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f402f7-0491-4d80-b0d6-a746821880ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4\" (UID: \"86f402f7-0491-4d80-b0d6-a746821880ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"
Sep 10 00:18:11.858883 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:11.858848    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"
Sep 10 00:18:12.136391 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:12.136359    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"]
Sep 10 00:18:12.138491 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:18:12.138467    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f402f7_0491_4d80_b0d6_a746821880ff.slice/crio-96ebbd1cad0bded331cf17ba4dfb5ccee7335818d4c0ec2560b199d445f083a6 WatchSource:0}: Error finding container 96ebbd1cad0bded331cf17ba4dfb5ccee7335818d4c0ec2560b199d445f083a6: Status 404 returned error can't find the container with id 96ebbd1cad0bded331cf17ba4dfb5ccee7335818d4c0ec2560b199d445f083a6
Sep 10 00:18:12.193370 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:12.193334    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4" event={"ID":"86f402f7-0491-4d80-b0d6-a746821880ff","Type":"ContainerStarted","Data":"96ebbd1cad0bded331cf17ba4dfb5ccee7335818d4c0ec2560b199d445f083a6"}
Sep 10 00:18:13.201618 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:13.201584    2556 generic.go:356] "Generic (PLEG): container finished" podID="86f402f7-0491-4d80-b0d6-a746821880ff" containerID="ec3df29ed66a9cda033c7e2e4efc0217f981bd7b8286cda60903eb4a3d934314" exitCode=0
Sep 10 00:18:13.202292 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:13.201628    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4" event={"ID":"86f402f7-0491-4d80-b0d6-a746821880ff","Type":"ContainerDied","Data":"ec3df29ed66a9cda033c7e2e4efc0217f981bd7b8286cda60903eb4a3d934314"}
Sep 10 00:18:14.359875 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:14.359852    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"
Sep 10 00:18:14.529665 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:14.529578    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f402f7-0491-4d80-b0d6-a746821880ff-maas-api-service-ca\") pod \"86f402f7-0491-4d80-b0d6-a746821880ff\" (UID: \"86f402f7-0491-4d80-b0d6-a746821880ff\") "
Sep 10 00:18:14.529944 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:14.529922    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f402f7-0491-4d80-b0d6-a746821880ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86f402f7-0491-4d80-b0d6-a746821880ff" (UID: "86f402f7-0491-4d80-b0d6-a746821880ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:18:14.630622 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:14.630585    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f402f7-0491-4d80-b0d6-a746821880ff-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:15.217147 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:15.217115    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"
Sep 10 00:18:15.217147 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:15.217131    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4" event={"ID":"86f402f7-0491-4d80-b0d6-a746821880ff","Type":"ContainerDied","Data":"96ebbd1cad0bded331cf17ba4dfb5ccee7335818d4c0ec2560b199d445f083a6"}
Sep 10 00:18:15.217353 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:15.217180    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96ebbd1cad0bded331cf17ba4dfb5ccee7335818d4c0ec2560b199d445f083a6"
Sep 10 00:18:21.964724 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:21.964683    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk"]
Sep 10 00:18:21.967436 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:21.967406    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-870d44-29816655-n9fpk"]
Sep 10 00:18:22.022228 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.022190    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"]
Sep 10 00:18:22.022514 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.022461    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph" podUID="1c9957ab-417f-4b86-8da4-d9b3480ff1ed" containerName="payload-processing" containerID="cri-o://786cfa1414ce85a309f17b8e8be7a95c47e4407cf307458bb798d76a64a898c4" gracePeriod=30
Sep 10 00:18:22.040690 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.040663    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"]
Sep 10 00:18:22.040891 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.040870    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2" podUID="1263ec79-e424-41dd-abaf-63445d26f97a" containerName="payload-pre-processing" containerID="cri-o://38bd695d373ed25390525763a37eb7b0d2a43f1b8ce5aad1642e7f473fdbfd0b" gracePeriod=30
Sep 10 00:18:22.243780 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.243697    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"]
Sep 10 00:18:22.247343 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.247319    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-870d44-p4mz4"]
Sep 10 00:18:22.277763 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.277733    2556 generic.go:356] "Generic (PLEG): container finished" podID="1263ec79-e424-41dd-abaf-63445d26f97a" containerID="38bd695d373ed25390525763a37eb7b0d2a43f1b8ce5aad1642e7f473fdbfd0b" exitCode=0
Sep 10 00:18:22.280131 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.280041    2556 generic.go:356] "Generic (PLEG): container finished" podID="1c9957ab-417f-4b86-8da4-d9b3480ff1ed" containerID="786cfa1414ce85a309f17b8e8be7a95c47e4407cf307458bb798d76a64a898c4" exitCode=0
Sep 10 00:18:22.281942 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.281919    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f402f7-0491-4d80-b0d6-a746821880ff" path="/var/lib/kubelet/pods/86f402f7-0491-4d80-b0d6-a746821880ff/volumes"
Sep 10 00:18:22.282426 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.282408    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4c52e85-fe8c-4539-a6db-e6e9e2448465" path="/var/lib/kubelet/pods/c4c52e85-fe8c-4539-a6db-e6e9e2448465/volumes"
Sep 10 00:18:22.282651 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.282635    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2" event={"ID":"1263ec79-e424-41dd-abaf-63445d26f97a","Type":"ContainerDied","Data":"38bd695d373ed25390525763a37eb7b0d2a43f1b8ce5aad1642e7f473fdbfd0b"}
Sep 10 00:18:22.282706 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.282675    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph" event={"ID":"1c9957ab-417f-4b86-8da4-d9b3480ff1ed","Type":"ContainerDied","Data":"786cfa1414ce85a309f17b8e8be7a95c47e4407cf307458bb798d76a64a898c4"}
Sep 10 00:18:22.306075 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.306037    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:18:22.326797 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.326776    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:18:22.407141 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.407109    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l46z\" (UniqueName: \"kubernetes.io/projected/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-kube-api-access-2l46z\") pod \"1c9957ab-417f-4b86-8da4-d9b3480ff1ed\" (UID: \"1c9957ab-417f-4b86-8da4-d9b3480ff1ed\") "
Sep 10 00:18:22.407349 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.407187    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-plugins-config-volume\") pod \"1c9957ab-417f-4b86-8da4-d9b3480ff1ed\" (UID: \"1c9957ab-417f-4b86-8da4-d9b3480ff1ed\") "
Sep 10 00:18:22.407640 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.407607    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c9957ab-417f-4b86-8da4-d9b3480ff1ed" (UID: "1c9957ab-417f-4b86-8da4-d9b3480ff1ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:18:22.409249 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.409224    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-kube-api-access-2l46z" (OuterVolumeSpecName: "kube-api-access-2l46z") pod "1c9957ab-417f-4b86-8da4-d9b3480ff1ed" (UID: "1c9957ab-417f-4b86-8da4-d9b3480ff1ed"). InnerVolumeSpecName "kube-api-access-2l46z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:22.509010 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.508922    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hblf\" (UniqueName: \"kubernetes.io/projected/1263ec79-e424-41dd-abaf-63445d26f97a-kube-api-access-6hblf\") pod \"1263ec79-e424-41dd-abaf-63445d26f97a\" (UID: \"1263ec79-e424-41dd-abaf-63445d26f97a\") "
Sep 10 00:18:22.509010 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.509002    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1263ec79-e424-41dd-abaf-63445d26f97a-plugins-config-volume\") pod \"1263ec79-e424-41dd-abaf-63445d26f97a\" (UID: \"1263ec79-e424-41dd-abaf-63445d26f97a\") "
Sep 10 00:18:22.509268 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.509257    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:22.509338 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.509268    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l46z\" (UniqueName: \"kubernetes.io/projected/1c9957ab-417f-4b86-8da4-d9b3480ff1ed-kube-api-access-2l46z\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:22.509440 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.509416    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1263ec79-e424-41dd-abaf-63445d26f97a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1263ec79-e424-41dd-abaf-63445d26f97a" (UID: "1263ec79-e424-41dd-abaf-63445d26f97a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:18:22.510961 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.510940    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1263ec79-e424-41dd-abaf-63445d26f97a-kube-api-access-6hblf" (OuterVolumeSpecName: "kube-api-access-6hblf") pod "1263ec79-e424-41dd-abaf-63445d26f97a" (UID: "1263ec79-e424-41dd-abaf-63445d26f97a"). InnerVolumeSpecName "kube-api-access-6hblf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:22.610472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.610440    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1263ec79-e424-41dd-abaf-63445d26f97a-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:22.610472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:22.610466    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hblf\" (UniqueName: \"kubernetes.io/projected/1263ec79-e424-41dd-abaf-63445d26f97a-kube-api-access-6hblf\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:23.287368 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:23.287334    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"
Sep 10 00:18:23.287816 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:23.287338    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2" event={"ID":"1263ec79-e424-41dd-abaf-63445d26f97a","Type":"ContainerDied","Data":"1a6fb5b2b82fa477f4a9eb2eb33bae8830461d473bd46c3a8c5ce89cea2eb7bd"}
Sep 10 00:18:23.287816 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:23.287469    2556 scope.go:117] "RemoveContainer" containerID="38bd695d373ed25390525763a37eb7b0d2a43f1b8ce5aad1642e7f473fdbfd0b"
Sep 10 00:18:23.288856 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:23.288823    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"
Sep 10 00:18:23.288938 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:23.288859    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph" event={"ID":"1c9957ab-417f-4b86-8da4-d9b3480ff1ed","Type":"ContainerDied","Data":"ccc110f488dc14502ee874aad9679d68e1b04dbbc8381f43c8183e776120c3d3"}
Sep 10 00:18:23.296890 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:23.296772    2556 scope.go:117] "RemoveContainer" containerID="786cfa1414ce85a309f17b8e8be7a95c47e4407cf307458bb798d76a64a898c4"
Sep 10 00:18:23.311458 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:23.311436    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"]
Sep 10 00:18:23.316906 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:23.316886    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-870d44-5d858486c8-jwqw2"]
Sep 10 00:18:23.325820 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:23.325800    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"]
Sep 10 00:18:23.327760 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:23.327743    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-870d44-5699647545-6p2ph"]
Sep 10 00:18:24.269427 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:24.269394    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1263ec79-e424-41dd-abaf-63445d26f97a" path="/var/lib/kubelet/pods/1263ec79-e424-41dd-abaf-63445d26f97a/volumes"
Sep 10 00:18:24.269795 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:24.269781    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c9957ab-417f-4b86-8da4-d9b3480ff1ed" path="/var/lib/kubelet/pods/1c9957ab-417f-4b86-8da4-d9b3480ff1ed/volumes"
Sep 10 00:18:33.265284 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.265195    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"]
Sep 10 00:18:33.266844 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.266816    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c9957ab-417f-4b86-8da4-d9b3480ff1ed" containerName="payload-processing"
Sep 10 00:18:33.266992 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.266980    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c9957ab-417f-4b86-8da4-d9b3480ff1ed" containerName="payload-processing"
Sep 10 00:18:33.267129 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.267115    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1263ec79-e424-41dd-abaf-63445d26f97a" containerName="payload-pre-processing"
Sep 10 00:18:33.267219 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.267210    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1263ec79-e424-41dd-abaf-63445d26f97a" containerName="payload-pre-processing"
Sep 10 00:18:33.267353 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.267344    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f402f7-0491-4d80-b0d6-a746821880ff" containerName="revoke-keys"
Sep 10 00:18:33.267419 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.267411    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f402f7-0491-4d80-b0d6-a746821880ff" containerName="revoke-keys"
Sep 10 00:18:33.267745 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.267730    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1263ec79-e424-41dd-abaf-63445d26f97a" containerName="payload-pre-processing"
Sep 10 00:18:33.267846 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.267836    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f402f7-0491-4d80-b0d6-a746821880ff" containerName="revoke-keys"
Sep 10 00:18:33.267920 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.267911    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c9957ab-417f-4b86-8da4-d9b3480ff1ed" containerName="payload-processing"
Sep 10 00:18:33.276984 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.276444    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"
Sep 10 00:18:33.279608 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.278971    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:18:33.279608 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.279321    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:18:33.279608 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.279407    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"]
Sep 10 00:18:33.314443 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.314402    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b4ce84-7770-4b43-a5b8-a012c53b7dba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf\" (UID: \"68b4ce84-7770-4b43-a5b8-a012c53b7dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"
Sep 10 00:18:33.415965 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.415930    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b4ce84-7770-4b43-a5b8-a012c53b7dba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf\" (UID: \"68b4ce84-7770-4b43-a5b8-a012c53b7dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"
Sep 10 00:18:33.416580 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.416560    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b4ce84-7770-4b43-a5b8-a012c53b7dba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf\" (UID: \"68b4ce84-7770-4b43-a5b8-a012c53b7dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"
Sep 10 00:18:33.589569 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.589482    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"
Sep 10 00:18:33.861717 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:33.861690    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"]
Sep 10 00:18:33.863320 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:18:33.863294    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b4ce84_7770_4b43_a5b8_a012c53b7dba.slice/crio-62e91c3c358b3e0fa26cf27f9f54696accd049defd10162cac4819a718ea5590 WatchSource:0}: Error finding container 62e91c3c358b3e0fa26cf27f9f54696accd049defd10162cac4819a718ea5590: Status 404 returned error can't find the container with id 62e91c3c358b3e0fa26cf27f9f54696accd049defd10162cac4819a718ea5590
Sep 10 00:18:34.356582 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:34.356554    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf" event={"ID":"68b4ce84-7770-4b43-a5b8-a012c53b7dba","Type":"ContainerStarted","Data":"fd71361ac7905024110105ac961ef3ce1ee4355dc0d0d39a7daa6c71e76bba83"}
Sep 10 00:18:34.356954 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:34.356604    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf" event={"ID":"68b4ce84-7770-4b43-a5b8-a012c53b7dba","Type":"ContainerStarted","Data":"62e91c3c358b3e0fa26cf27f9f54696accd049defd10162cac4819a718ea5590"}
Sep 10 00:18:34.371670 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:34.371600    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf" podStartSLOduration=1.371585452 podStartE2EDuration="1.371585452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:18:34.369799509 +0000 UTC m=+1794.620935552" watchObservedRunningTime="2026-09-10 00:18:34.371585452 +0000 UTC m=+1794.622721563"
Sep 10 00:18:35.364625 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:35.364591    2556 generic.go:356] "Generic (PLEG): container finished" podID="68b4ce84-7770-4b43-a5b8-a012c53b7dba" containerID="fd71361ac7905024110105ac961ef3ce1ee4355dc0d0d39a7daa6c71e76bba83" exitCode=0
Sep 10 00:18:35.364993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:35.364674    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf" event={"ID":"68b4ce84-7770-4b43-a5b8-a012c53b7dba","Type":"ContainerDied","Data":"fd71361ac7905024110105ac961ef3ce1ee4355dc0d0d39a7daa6c71e76bba83"}
Sep 10 00:18:36.507171 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:36.507148    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"
Sep 10 00:18:36.545785 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:36.545759    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b4ce84-7770-4b43-a5b8-a012c53b7dba-maas-api-service-ca\") pod \"68b4ce84-7770-4b43-a5b8-a012c53b7dba\" (UID: \"68b4ce84-7770-4b43-a5b8-a012c53b7dba\") "
Sep 10 00:18:36.546164 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:36.546144    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b4ce84-7770-4b43-a5b8-a012c53b7dba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68b4ce84-7770-4b43-a5b8-a012c53b7dba" (UID: "68b4ce84-7770-4b43-a5b8-a012c53b7dba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:18:36.646634 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:36.646552    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b4ce84-7770-4b43-a5b8-a012c53b7dba-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:37.378209 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:37.378171    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"
Sep 10 00:18:37.378209 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:37.378176    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf" event={"ID":"68b4ce84-7770-4b43-a5b8-a012c53b7dba","Type":"ContainerDied","Data":"62e91c3c358b3e0fa26cf27f9f54696accd049defd10162cac4819a718ea5590"}
Sep 10 00:18:37.378448 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:37.378230    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62e91c3c358b3e0fa26cf27f9f54696accd049defd10162cac4819a718ea5590"
Sep 10 00:18:38.146977 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:38.146940    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"]
Sep 10 00:18:38.154684 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:38.154657    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"]
Sep 10 00:18:38.162864 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:38.162839    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"]
Sep 10 00:18:38.174233 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:38.174210    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"]
Sep 10 00:18:38.181988 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:38.181962    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"]
Sep 10 00:18:48.880075 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:48.879973    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"]
Sep 10 00:18:48.880611 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:48.880357    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw" podUID="df6bb0d0-df23-4a5e-ae76-13482129b996" containerName="payload-processing" containerID="cri-o://08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e" gracePeriod=30
Sep 10 00:18:48.886034 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:48.886009    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"]
Sep 10 00:18:48.886437 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:48.886380    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl" podUID="a06d7cda-d92c-4ad0-83d3-34177df92834" containerName="payload-pre-processing" containerID="cri-o://2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092" gracePeriod=30
Sep 10 00:18:49.037134 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.037080    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"]
Sep 10 00:18:49.041176 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.041135    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-01b7d5-fkjlf"]
Sep 10 00:18:49.137691 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.137640    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:18:49.168384 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.168363    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:18:49.259232 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.259208    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df6bb0d0-df23-4a5e-ae76-13482129b996-plugins-config-volume\") pod \"df6bb0d0-df23-4a5e-ae76-13482129b996\" (UID: \"df6bb0d0-df23-4a5e-ae76-13482129b996\") "
Sep 10 00:18:49.259354 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.259271    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slmrg\" (UniqueName: \"kubernetes.io/projected/df6bb0d0-df23-4a5e-ae76-13482129b996-kube-api-access-slmrg\") pod \"df6bb0d0-df23-4a5e-ae76-13482129b996\" (UID: \"df6bb0d0-df23-4a5e-ae76-13482129b996\") "
Sep 10 00:18:49.259354 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.259311    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhlr6\" (UniqueName: \"kubernetes.io/projected/a06d7cda-d92c-4ad0-83d3-34177df92834-kube-api-access-vhlr6\") pod \"a06d7cda-d92c-4ad0-83d3-34177df92834\" (UID: \"a06d7cda-d92c-4ad0-83d3-34177df92834\") "
Sep 10 00:18:49.259354 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.259333    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06d7cda-d92c-4ad0-83d3-34177df92834-plugins-config-volume\") pod \"a06d7cda-d92c-4ad0-83d3-34177df92834\" (UID: \"a06d7cda-d92c-4ad0-83d3-34177df92834\") "
Sep 10 00:18:49.259721 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.259693    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6bb0d0-df23-4a5e-ae76-13482129b996-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df6bb0d0-df23-4a5e-ae76-13482129b996" (UID: "df6bb0d0-df23-4a5e-ae76-13482129b996"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:18:49.259788 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.259763    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06d7cda-d92c-4ad0-83d3-34177df92834-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a06d7cda-d92c-4ad0-83d3-34177df92834" (UID: "a06d7cda-d92c-4ad0-83d3-34177df92834"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:18:49.261480 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.261460    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06d7cda-d92c-4ad0-83d3-34177df92834-kube-api-access-vhlr6" (OuterVolumeSpecName: "kube-api-access-vhlr6") pod "a06d7cda-d92c-4ad0-83d3-34177df92834" (UID: "a06d7cda-d92c-4ad0-83d3-34177df92834"). InnerVolumeSpecName "kube-api-access-vhlr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:49.261544 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.261502    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6bb0d0-df23-4a5e-ae76-13482129b996-kube-api-access-slmrg" (OuterVolumeSpecName: "kube-api-access-slmrg") pod "df6bb0d0-df23-4a5e-ae76-13482129b996" (UID: "df6bb0d0-df23-4a5e-ae76-13482129b996"). InnerVolumeSpecName "kube-api-access-slmrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:49.360110 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.360083    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slmrg\" (UniqueName: \"kubernetes.io/projected/df6bb0d0-df23-4a5e-ae76-13482129b996-kube-api-access-slmrg\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:49.360110 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.360106    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhlr6\" (UniqueName: \"kubernetes.io/projected/a06d7cda-d92c-4ad0-83d3-34177df92834-kube-api-access-vhlr6\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:49.360110 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.360115    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06d7cda-d92c-4ad0-83d3-34177df92834-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:49.360286 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.360125    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df6bb0d0-df23-4a5e-ae76-13482129b996-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:49.452041 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.452010    2556 generic.go:356] "Generic (PLEG): container finished" podID="a06d7cda-d92c-4ad0-83d3-34177df92834" containerID="2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092" exitCode=0
Sep 10 00:18:49.452175 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.452084    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"
Sep 10 00:18:49.452175 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.452102    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl" event={"ID":"a06d7cda-d92c-4ad0-83d3-34177df92834","Type":"ContainerDied","Data":"2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092"}
Sep 10 00:18:49.452175 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.452152    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl" event={"ID":"a06d7cda-d92c-4ad0-83d3-34177df92834","Type":"ContainerDied","Data":"56a39db71d4baad5f3d3e00fafe8800260e6f02fe3396df8094f48b8126a6b3a"}
Sep 10 00:18:49.452175 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.452167    2556 scope.go:117] "RemoveContainer" containerID="2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092"
Sep 10 00:18:49.453791 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.453760    2556 generic.go:356] "Generic (PLEG): container finished" podID="df6bb0d0-df23-4a5e-ae76-13482129b996" containerID="08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e" exitCode=0
Sep 10 00:18:49.453903 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.453812    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"
Sep 10 00:18:49.453903 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.453851    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw" event={"ID":"df6bb0d0-df23-4a5e-ae76-13482129b996","Type":"ContainerDied","Data":"08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e"}
Sep 10 00:18:49.454006 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.453918    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw" event={"ID":"df6bb0d0-df23-4a5e-ae76-13482129b996","Type":"ContainerDied","Data":"67fed14a7d2a0d18183672198d253b324f279223e40fee7dcd1e5c1192d0eda2"}
Sep 10 00:18:49.462141 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.462122    2556 scope.go:117] "RemoveContainer" containerID="2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092"
Sep 10 00:18:49.462457 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:18:49.462433    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092\": container with ID starting with 2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092 not found: ID does not exist" containerID="2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092"
Sep 10 00:18:49.462537 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.462469    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092"} err="failed to get container status \"2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092\": rpc error: code = NotFound desc = could not find container \"2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092\": container with ID starting with 2587e953a206c1fbc16313efc040931ffb013b46b7775f60aa5217daed082092 not found: ID does not exist"
Sep 10 00:18:49.462537 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.462494    2556 scope.go:117] "RemoveContainer" containerID="08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e"
Sep 10 00:18:49.471697 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.471679    2556 scope.go:117] "RemoveContainer" containerID="08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e"
Sep 10 00:18:49.471947 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:18:49.471931    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e\": container with ID starting with 08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e not found: ID does not exist" containerID="08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e"
Sep 10 00:18:49.471996 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.471952    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e"} err="failed to get container status \"08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e\": rpc error: code = NotFound desc = could not find container \"08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e\": container with ID starting with 08ed4a67e44b3774e3fd18190d5d9d5c218a926c36e69022383f08feafd5733e not found: ID does not exist"
Sep 10 00:18:49.477867 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.477846    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"]
Sep 10 00:18:49.481486 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.481466    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-01b7d5-757b98865d-vxbxl"]
Sep 10 00:18:49.490591 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.490571    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"]
Sep 10 00:18:49.493606 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:49.493588    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-01b7d5-f985bff5c-4mxmw"]
Sep 10 00:18:50.268984 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:50.268942    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68b4ce84-7770-4b43-a5b8-a012c53b7dba" path="/var/lib/kubelet/pods/68b4ce84-7770-4b43-a5b8-a012c53b7dba/volumes"
Sep 10 00:18:50.269336 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:50.269324    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a06d7cda-d92c-4ad0-83d3-34177df92834" path="/var/lib/kubelet/pods/a06d7cda-d92c-4ad0-83d3-34177df92834/volumes"
Sep 10 00:18:50.269662 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:50.269648    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df6bb0d0-df23-4a5e-ae76-13482129b996" path="/var/lib/kubelet/pods/df6bb0d0-df23-4a5e-ae76-13482129b996/volumes"
Sep 10 00:18:53.168604 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.168562    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb" podUID="1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" containerName="main" containerID="cri-o://267e6f543aa7d7a8257cd87328af61307f05bae6e1f81cca3f541b25c0dbc9a1" gracePeriod=45
Sep 10 00:18:53.174906 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.174871    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6" podUID="166c136a-d51b-4ac0-87ca-eb685101acd3" containerName="main" containerID="cri-o://a0a24e0d5fa9a232bae253a2f4ef53f7d154da3db7c6a9aed6a602ccf7d8aa6f" gracePeriod=45
Sep 10 00:18:53.184040 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.183996    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx" podUID="2993af8b-6a57-4b6c-8046-48297c8323a4" containerName="main" containerID="cri-o://6a64fe5098a4aa80e3a3e189d2eb21e8e639a681ad00b1a0a898b922d029ce99" gracePeriod=45
Sep 10 00:18:53.195085 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.195031    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k" podUID="bf09d0d1-b106-49ad-ad92-6724c0d67aa1" containerName="main" containerID="cri-o://3a5c7939aa34794cc24104ce1b5157fbad9ad6c1da713d9abe16c71afa6609d0" gracePeriod=45
Sep 10 00:18:53.201119 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.201090    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh" podUID="d9225853-6085-4389-88b7-2bca95a12e4a" containerName="main" containerID="cri-o://5f6ce2161791af0a97a3e817770a9e5e2868eaa6ea3d4035018314fd8adfb430" gracePeriod=45
Sep 10 00:18:53.483294 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.483263    2556 generic.go:356] "Generic (PLEG): container finished" podID="d9225853-6085-4389-88b7-2bca95a12e4a" containerID="5f6ce2161791af0a97a3e817770a9e5e2868eaa6ea3d4035018314fd8adfb430" exitCode=0
Sep 10 00:18:53.483457 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.483339    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh" event={"ID":"d9225853-6085-4389-88b7-2bca95a12e4a","Type":"ContainerDied","Data":"5f6ce2161791af0a97a3e817770a9e5e2868eaa6ea3d4035018314fd8adfb430"}
Sep 10 00:18:53.485164 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.485142    2556 generic.go:356] "Generic (PLEG): container finished" podID="166c136a-d51b-4ac0-87ca-eb685101acd3" containerID="a0a24e0d5fa9a232bae253a2f4ef53f7d154da3db7c6a9aed6a602ccf7d8aa6f" exitCode=0
Sep 10 00:18:53.485291 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.485213    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6" event={"ID":"166c136a-d51b-4ac0-87ca-eb685101acd3","Type":"ContainerDied","Data":"a0a24e0d5fa9a232bae253a2f4ef53f7d154da3db7c6a9aed6a602ccf7d8aa6f"}
Sep 10 00:18:53.486945 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.486926    2556 generic.go:356] "Generic (PLEG): container finished" podID="bf09d0d1-b106-49ad-ad92-6724c0d67aa1" containerID="3a5c7939aa34794cc24104ce1b5157fbad9ad6c1da713d9abe16c71afa6609d0" exitCode=0
Sep 10 00:18:53.487048 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.487002    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k" event={"ID":"bf09d0d1-b106-49ad-ad92-6724c0d67aa1","Type":"ContainerDied","Data":"3a5c7939aa34794cc24104ce1b5157fbad9ad6c1da713d9abe16c71afa6609d0"}
Sep 10 00:18:53.488769 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.488750    2556 generic.go:356] "Generic (PLEG): container finished" podID="2993af8b-6a57-4b6c-8046-48297c8323a4" containerID="6a64fe5098a4aa80e3a3e189d2eb21e8e639a681ad00b1a0a898b922d029ce99" exitCode=0
Sep 10 00:18:53.488881 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.488792    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx" event={"ID":"2993af8b-6a57-4b6c-8046-48297c8323a4","Type":"ContainerDied","Data":"6a64fe5098a4aa80e3a3e189d2eb21e8e639a681ad00b1a0a898b922d029ce99"}
Sep 10 00:18:53.491561 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.490352    2556 generic.go:356] "Generic (PLEG): container finished" podID="1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" containerID="267e6f543aa7d7a8257cd87328af61307f05bae6e1f81cca3f541b25c0dbc9a1" exitCode=0
Sep 10 00:18:53.491561 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.490396    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb" event={"ID":"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1","Type":"ContainerDied","Data":"267e6f543aa7d7a8257cd87328af61307f05bae6e1f81cca3f541b25c0dbc9a1"}
Sep 10 00:18:53.491561 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.490424    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb" event={"ID":"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1","Type":"ContainerDied","Data":"951605e8e1981aed539c6a9d6ec9f471c009c1d1e8a6cab77afa14ed2ed22832"}
Sep 10 00:18:53.491561 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.490440    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="951605e8e1981aed539c6a9d6ec9f471c009c1d1e8a6cab77afa14ed2ed22832"
Sep 10 00:18:53.510189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.510166    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:18:53.523428 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.523407    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:18:53.542715 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.542666    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:18:53.546405 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.546384    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:18:53.550603 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.550581    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:18:53.598508 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598480    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfx2b\" (UniqueName: \"kubernetes.io/projected/2993af8b-6a57-4b6c-8046-48297c8323a4-kube-api-access-xfx2b\") pod \"2993af8b-6a57-4b6c-8046-48297c8323a4\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") "
Sep 10 00:18:53.598667 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598528    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tmp-dir\") pod \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") "
Sep 10 00:18:53.598667 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598603    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2993af8b-6a57-4b6c-8046-48297c8323a4-tls-certs\") pod \"2993af8b-6a57-4b6c-8046-48297c8323a4\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") "
Sep 10 00:18:53.598667 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598645    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-dshm\") pod \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") "
Sep 10 00:18:53.598834 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598674    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-dshm\") pod \"2993af8b-6a57-4b6c-8046-48297c8323a4\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") "
Sep 10 00:18:53.598834 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598716    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl88m\" (UniqueName: \"kubernetes.io/projected/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-kube-api-access-jl88m\") pod \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") "
Sep 10 00:18:53.598834 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598775    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-home\") pod \"2993af8b-6a57-4b6c-8046-48297c8323a4\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") "
Sep 10 00:18:53.598834 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598812    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-home\") pod \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") "
Sep 10 00:18:53.599041 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598835    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" (UID: "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.599041 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598846    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-model-cache\") pod \"2993af8b-6a57-4b6c-8046-48297c8323a4\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") "
Sep 10 00:18:53.599041 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598906    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-tmp-dir\") pod \"2993af8b-6a57-4b6c-8046-48297c8323a4\" (UID: \"2993af8b-6a57-4b6c-8046-48297c8323a4\") "
Sep 10 00:18:53.599041 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.598976    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tls-certs\") pod \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") "
Sep 10 00:18:53.599041 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.599035    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-model-cache\") pod \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\" (UID: \"1f455ae5-9cdb-4a87-834b-4f1bb4c499d1\") "
Sep 10 00:18:53.599420 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.599397    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "2993af8b-6a57-4b6c-8046-48297c8323a4" (UID: "2993af8b-6a57-4b6c-8046-48297c8323a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.599498 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.599463    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2993af8b-6a57-4b6c-8046-48297c8323a4" (UID: "2993af8b-6a57-4b6c-8046-48297c8323a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.599556 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.599525    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.599556 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.599546    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.599670 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.599606    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-home" (OuterVolumeSpecName: "home") pod "2993af8b-6a57-4b6c-8046-48297c8323a4" (UID: "2993af8b-6a57-4b6c-8046-48297c8323a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.599732 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.599714    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-home" (OuterVolumeSpecName: "home") pod "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" (UID: "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.600258 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.600232    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" (UID: "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.601385 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.601359    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2993af8b-6a57-4b6c-8046-48297c8323a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2993af8b-6a57-4b6c-8046-48297c8323a4" (UID: "2993af8b-6a57-4b6c-8046-48297c8323a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:18:53.601794 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.601766    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2993af8b-6a57-4b6c-8046-48297c8323a4-kube-api-access-xfx2b" (OuterVolumeSpecName: "kube-api-access-xfx2b") pod "2993af8b-6a57-4b6c-8046-48297c8323a4" (UID: "2993af8b-6a57-4b6c-8046-48297c8323a4"). InnerVolumeSpecName "kube-api-access-xfx2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:53.602072 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.602035    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-dshm" (OuterVolumeSpecName: "dshm") pod "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" (UID: "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.602235 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.602186    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" (UID: "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:18:53.602318 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.602294    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-kube-api-access-jl88m" (OuterVolumeSpecName: "kube-api-access-jl88m") pod "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" (UID: "1f455ae5-9cdb-4a87-834b-4f1bb4c499d1"). InnerVolumeSpecName "kube-api-access-jl88m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:53.602406 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.602383    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-dshm" (OuterVolumeSpecName: "dshm") pod "2993af8b-6a57-4b6c-8046-48297c8323a4" (UID: "2993af8b-6a57-4b6c-8046-48297c8323a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.699911 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.699832    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/166c136a-d51b-4ac0-87ca-eb685101acd3-tls-certs\") pod \"166c136a-d51b-4ac0-87ca-eb685101acd3\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") "
Sep 10 00:18:53.699911 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.699875    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9225853-6085-4389-88b7-2bca95a12e4a-tls-certs\") pod \"d9225853-6085-4389-88b7-2bca95a12e4a\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") "
Sep 10 00:18:53.700157 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.699910    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvrlr\" (UniqueName: \"kubernetes.io/projected/d9225853-6085-4389-88b7-2bca95a12e4a-kube-api-access-vvrlr\") pod \"d9225853-6085-4389-88b7-2bca95a12e4a\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") "
Sep 10 00:18:53.700157 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.699945    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-model-cache\") pod \"d9225853-6085-4389-88b7-2bca95a12e4a\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") "
Sep 10 00:18:53.700157 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.699976    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-model-cache\") pod \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") "
Sep 10 00:18:53.700157 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.699997    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-dshm\") pod \"d9225853-6085-4389-88b7-2bca95a12e4a\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") "
Sep 10 00:18:53.700157 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700017    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-tmp-dir\") pod \"d9225853-6085-4389-88b7-2bca95a12e4a\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") "
Sep 10 00:18:53.700157 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700119    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-dshm\") pod \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") "
Sep 10 00:18:53.700157 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700146    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-home\") pod \"d9225853-6085-4389-88b7-2bca95a12e4a\" (UID: \"d9225853-6085-4389-88b7-2bca95a12e4a\") "
Sep 10 00:18:53.700513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700206    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb5bg\" (UniqueName: \"kubernetes.io/projected/166c136a-d51b-4ac0-87ca-eb685101acd3-kube-api-access-sb5bg\") pod \"166c136a-d51b-4ac0-87ca-eb685101acd3\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") "
Sep 10 00:18:53.700513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700257    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-model-cache\") pod \"166c136a-d51b-4ac0-87ca-eb685101acd3\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") "
Sep 10 00:18:53.700513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700252    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf09d0d1-b106-49ad-ad92-6724c0d67aa1" (UID: "bf09d0d1-b106-49ad-ad92-6724c0d67aa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.700513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700256    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9225853-6085-4389-88b7-2bca95a12e4a" (UID: "d9225853-6085-4389-88b7-2bca95a12e4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.700513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700284    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-home\") pod \"166c136a-d51b-4ac0-87ca-eb685101acd3\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") "
Sep 10 00:18:53.700513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700335    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-dshm\") pod \"166c136a-d51b-4ac0-87ca-eb685101acd3\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") "
Sep 10 00:18:53.700513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700375    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mjlk\" (UniqueName: \"kubernetes.io/projected/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-kube-api-access-8mjlk\") pod \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") "
Sep 10 00:18:53.700513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700479    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tmp-dir\") pod \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") "
Sep 10 00:18:53.700513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700506    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-home" (OuterVolumeSpecName: "home") pod "166c136a-d51b-4ac0-87ca-eb685101acd3" (UID: "166c136a-d51b-4ac0-87ca-eb685101acd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.700513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700517    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-home\") pod \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") "
Sep 10 00:18:53.701016 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700553    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-tmp-dir\") pod \"166c136a-d51b-4ac0-87ca-eb685101acd3\" (UID: \"166c136a-d51b-4ac0-87ca-eb685101acd3\") "
Sep 10 00:18:53.701016 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700590    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tls-certs\") pod \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\" (UID: \"bf09d0d1-b106-49ad-ad92-6724c0d67aa1\") "
Sep 10 00:18:53.701016 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.700684    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "166c136a-d51b-4ac0-87ca-eb685101acd3" (UID: "166c136a-d51b-4ac0-87ca-eb685101acd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.701016 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701003    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701021    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701034    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701048    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701082    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfx2b\" (UniqueName: \"kubernetes.io/projected/2993af8b-6a57-4b6c-8046-48297c8323a4-kube-api-access-xfx2b\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701098    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2993af8b-6a57-4b6c-8046-48297c8323a4-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701111    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701126    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701139    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl88m\" (UniqueName: \"kubernetes.io/projected/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-kube-api-access-jl88m\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701152    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701166    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701178    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701190    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701247 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701202    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2993af8b-6a57-4b6c-8046-48297c8323a4-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.701924 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701403    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9225853-6085-4389-88b7-2bca95a12e4a" (UID: "d9225853-6085-4389-88b7-2bca95a12e4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.701924 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.701568    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf09d0d1-b106-49ad-ad92-6724c0d67aa1" (UID: "bf09d0d1-b106-49ad-ad92-6724c0d67aa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.703185 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.703150    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9225853-6085-4389-88b7-2bca95a12e4a-kube-api-access-vvrlr" (OuterVolumeSpecName: "kube-api-access-vvrlr") pod "d9225853-6085-4389-88b7-2bca95a12e4a" (UID: "d9225853-6085-4389-88b7-2bca95a12e4a"). InnerVolumeSpecName "kube-api-access-vvrlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:53.703185 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.703170    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf09d0d1-b106-49ad-ad92-6724c0d67aa1" (UID: "bf09d0d1-b106-49ad-ad92-6724c0d67aa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:18:53.703365 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.703170    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166c136a-d51b-4ac0-87ca-eb685101acd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "166c136a-d51b-4ac0-87ca-eb685101acd3" (UID: "166c136a-d51b-4ac0-87ca-eb685101acd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:18:53.703365 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.703197    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9225853-6085-4389-88b7-2bca95a12e4a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9225853-6085-4389-88b7-2bca95a12e4a" (UID: "d9225853-6085-4389-88b7-2bca95a12e4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:18:53.703365 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.703197    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-dshm" (OuterVolumeSpecName: "dshm") pod "bf09d0d1-b106-49ad-ad92-6724c0d67aa1" (UID: "bf09d0d1-b106-49ad-ad92-6724c0d67aa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.703509 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.703400    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "166c136a-d51b-4ac0-87ca-eb685101acd3" (UID: "166c136a-d51b-4ac0-87ca-eb685101acd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.703509 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.703428    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-home" (OuterVolumeSpecName: "home") pod "d9225853-6085-4389-88b7-2bca95a12e4a" (UID: "d9225853-6085-4389-88b7-2bca95a12e4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.703509 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.703445    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166c136a-d51b-4ac0-87ca-eb685101acd3-kube-api-access-sb5bg" (OuterVolumeSpecName: "kube-api-access-sb5bg") pod "166c136a-d51b-4ac0-87ca-eb685101acd3" (UID: "166c136a-d51b-4ac0-87ca-eb685101acd3"). InnerVolumeSpecName "kube-api-access-sb5bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:53.703509 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.703491    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-dshm" (OuterVolumeSpecName: "dshm") pod "166c136a-d51b-4ac0-87ca-eb685101acd3" (UID: "166c136a-d51b-4ac0-87ca-eb685101acd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.703509 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.703502    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-home" (OuterVolumeSpecName: "home") pod "bf09d0d1-b106-49ad-ad92-6724c0d67aa1" (UID: "bf09d0d1-b106-49ad-ad92-6724c0d67aa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.703818 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.703799    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-dshm" (OuterVolumeSpecName: "dshm") pod "d9225853-6085-4389-88b7-2bca95a12e4a" (UID: "d9225853-6085-4389-88b7-2bca95a12e4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:18:53.704857 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.704841    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-kube-api-access-8mjlk" (OuterVolumeSpecName: "kube-api-access-8mjlk") pod "bf09d0d1-b106-49ad-ad92-6724c0d67aa1" (UID: "bf09d0d1-b106-49ad-ad92-6724c0d67aa1"). InnerVolumeSpecName "kube-api-access-8mjlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:18:53.802630 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802610    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802630 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802630    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mjlk\" (UniqueName: \"kubernetes.io/projected/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-kube-api-access-8mjlk\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802642    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802650    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802658    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/166c136a-d51b-4ac0-87ca-eb685101acd3-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802666    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802679    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/166c136a-d51b-4ac0-87ca-eb685101acd3-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802689    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9225853-6085-4389-88b7-2bca95a12e4a-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802698    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvrlr\" (UniqueName: \"kubernetes.io/projected/d9225853-6085-4389-88b7-2bca95a12e4a-kube-api-access-vvrlr\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802705    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802717    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802728    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf09d0d1-b106-49ad-ad92-6724c0d67aa1-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802740    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9225853-6085-4389-88b7-2bca95a12e4a-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:53.802764 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:53.802752    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb5bg\" (UniqueName: \"kubernetes.io/projected/166c136a-d51b-4ac0-87ca-eb685101acd3-kube-api-access-sb5bg\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:18:54.500472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.500439    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"
Sep 10 00:18:54.500867 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.500494    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k" event={"ID":"bf09d0d1-b106-49ad-ad92-6724c0d67aa1","Type":"ContainerDied","Data":"a47f869b3a6c29f59647c360643a320c6ce5f0838b60359ccb2719ed78a13063"}
Sep 10 00:18:54.500867 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.500549    2556 scope.go:117] "RemoveContainer" containerID="3a5c7939aa34794cc24104ce1b5157fbad9ad6c1da713d9abe16c71afa6609d0"
Sep 10 00:18:54.502153 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.502045    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"
Sep 10 00:18:54.502153 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.502094    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx" event={"ID":"2993af8b-6a57-4b6c-8046-48297c8323a4","Type":"ContainerDied","Data":"fc7a34aea5760549238b53a1640140179ea3afbdb6a9cb4c7922356db4725140"}
Sep 10 00:18:54.505106 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.504749    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"
Sep 10 00:18:54.505106 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.504818    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh" event={"ID":"d9225853-6085-4389-88b7-2bca95a12e4a","Type":"ContainerDied","Data":"27499dc2694304610b10272ad917794ffac2ee22fb187a6f05b8f295d63a16e4"}
Sep 10 00:18:54.506920 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.506881    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6" event={"ID":"166c136a-d51b-4ac0-87ca-eb685101acd3","Type":"ContainerDied","Data":"71fa9d9844309561426a790c95c98cbc30c04ac80391bc761d475a6e759d826b"}
Sep 10 00:18:54.507035 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.506898    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"
Sep 10 00:18:54.507134 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.506902    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"
Sep 10 00:18:54.513531 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.513514    2556 scope.go:117] "RemoveContainer" containerID="6a64fe5098a4aa80e3a3e189d2eb21e8e639a681ad00b1a0a898b922d029ce99"
Sep 10 00:18:54.520096 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.520043    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"]
Sep 10 00:18:54.521986 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.521967    2556 scope.go:117] "RemoveContainer" containerID="5f6ce2161791af0a97a3e817770a9e5e2868eaa6ea3d4035018314fd8adfb430"
Sep 10 00:18:54.526643 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.526611    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/premium-w0-01b7d5-kserve-85fdf564bf-tmzxx"]
Sep 10 00:18:54.536993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.536979    2556 scope.go:117] "RemoveContainer" containerID="a0a24e0d5fa9a232bae253a2f4ef53f7d154da3db7c6a9aed6a602ccf7d8aa6f"
Sep 10 00:18:54.540676 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.540653    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"]
Sep 10 00:18:54.545641 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.545613    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/simulator-w0-01b7d5-kserve-5bdfc54f6d-zq88k"]
Sep 10 00:18:54.552724 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.552704    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"]
Sep 10 00:18:54.557917 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.557898    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/unconfigured-w0-01b7d5-kserve-76b45bb8dc-jvlhh"]
Sep 10 00:18:54.567327 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.567308    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"]
Sep 10 00:18:54.573797 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.573777    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/embedding-w0-01b7d5-kserve-5f9b789c49-zrnw6"]
Sep 10 00:18:54.584083 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.584049    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"]
Sep 10 00:18:54.587336 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:54.587314    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-01b7d5/distinct-w0-01b7d5-kserve-6cbd76fd68-npzlb"]
Sep 10 00:18:56.268049 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:56.268013    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="166c136a-d51b-4ac0-87ca-eb685101acd3" path="/var/lib/kubelet/pods/166c136a-d51b-4ac0-87ca-eb685101acd3/volumes"
Sep 10 00:18:56.268463 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:56.268445    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" path="/var/lib/kubelet/pods/1f455ae5-9cdb-4a87-834b-4f1bb4c499d1/volumes"
Sep 10 00:18:56.268804 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:56.268792    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2993af8b-6a57-4b6c-8046-48297c8323a4" path="/var/lib/kubelet/pods/2993af8b-6a57-4b6c-8046-48297c8323a4/volumes"
Sep 10 00:18:56.269476 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:56.269458    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf09d0d1-b106-49ad-ad92-6724c0d67aa1" path="/var/lib/kubelet/pods/bf09d0d1-b106-49ad-ad92-6724c0d67aa1/volumes"
Sep 10 00:18:56.269855 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:18:56.269841    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9225853-6085-4389-88b7-2bca95a12e4a" path="/var/lib/kubelet/pods/d9225853-6085-4389-88b7-2bca95a12e4a/volumes"
Sep 10 00:19:34.210275 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210235    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"]
Sep 10 00:19:34.210774 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210738    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b4ce84-7770-4b43-a5b8-a012c53b7dba" containerName="revoke-keys"
Sep 10 00:19:34.210774 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210752    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b4ce84-7770-4b43-a5b8-a012c53b7dba" containerName="revoke-keys"
Sep 10 00:19:34.210774 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210766    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" containerName="main"
Sep 10 00:19:34.210774 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210772    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" containerName="main"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210792    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2993af8b-6a57-4b6c-8046-48297c8323a4" containerName="main"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210798    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2993af8b-6a57-4b6c-8046-48297c8323a4" containerName="main"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210805    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="166c136a-d51b-4ac0-87ca-eb685101acd3" containerName="main"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210809    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="166c136a-d51b-4ac0-87ca-eb685101acd3" containerName="main"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210819    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9225853-6085-4389-88b7-2bca95a12e4a" containerName="main"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210824    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9225853-6085-4389-88b7-2bca95a12e4a" containerName="main"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210834    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6bb0d0-df23-4a5e-ae76-13482129b996" containerName="payload-processing"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210839    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6bb0d0-df23-4a5e-ae76-13482129b996" containerName="payload-processing"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210847    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf09d0d1-b106-49ad-ad92-6724c0d67aa1" containerName="main"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210853    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf09d0d1-b106-49ad-ad92-6724c0d67aa1" containerName="main"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210865    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a06d7cda-d92c-4ad0-83d3-34177df92834" containerName="payload-pre-processing"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210873    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a06d7cda-d92c-4ad0-83d3-34177df92834" containerName="payload-pre-processing"
Sep 10 00:19:34.210937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210939    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="68b4ce84-7770-4b43-a5b8-a012c53b7dba" containerName="revoke-keys"
Sep 10 00:19:34.211355 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210949    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6bb0d0-df23-4a5e-ae76-13482129b996" containerName="payload-processing"
Sep 10 00:19:34.211355 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210958    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2993af8b-6a57-4b6c-8046-48297c8323a4" containerName="main"
Sep 10 00:19:34.211355 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210965    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f455ae5-9cdb-4a87-834b-4f1bb4c499d1" containerName="main"
Sep 10 00:19:34.211355 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210971    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf09d0d1-b106-49ad-ad92-6724c0d67aa1" containerName="main"
Sep 10 00:19:34.211355 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210978    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="166c136a-d51b-4ac0-87ca-eb685101acd3" containerName="main"
Sep 10 00:19:34.211355 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210984    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9225853-6085-4389-88b7-2bca95a12e4a" containerName="main"
Sep 10 00:19:34.211355 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.210990    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a06d7cda-d92c-4ad0-83d3-34177df92834" containerName="payload-pre-processing"
Sep 10 00:19:34.214329 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.214315    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"
Sep 10 00:19:34.216584 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.216561    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:19:34.216709 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.216635    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:19:34.221988 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.221701    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"]
Sep 10 00:19:34.366200 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.366160    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b866f960-c15a-4242-ac05-e1bce0cc0c69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj\" (UID: \"b866f960-c15a-4242-ac05-e1bce0cc0c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"
Sep 10 00:19:34.467753 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.467670    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b866f960-c15a-4242-ac05-e1bce0cc0c69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj\" (UID: \"b866f960-c15a-4242-ac05-e1bce0cc0c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"
Sep 10 00:19:34.468307 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.468285    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b866f960-c15a-4242-ac05-e1bce0cc0c69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj\" (UID: \"b866f960-c15a-4242-ac05-e1bce0cc0c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"
Sep 10 00:19:34.526151 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.526127    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"
Sep 10 00:19:34.793725 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:34.793702    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"]
Sep 10 00:19:34.795806 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:19:34.795772    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb866f960_c15a_4242_ac05_e1bce0cc0c69.slice/crio-a209ec197f8d7433ac221b617bdb2a653604cf6041017731f142cb17fe193adc WatchSource:0}: Error finding container a209ec197f8d7433ac221b617bdb2a653604cf6041017731f142cb17fe193adc: Status 404 returned error can't find the container with id a209ec197f8d7433ac221b617bdb2a653604cf6041017731f142cb17fe193adc
Sep 10 00:19:35.753046 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:35.753014    2556 generic.go:356] "Generic (PLEG): container finished" podID="b866f960-c15a-4242-ac05-e1bce0cc0c69" containerID="11b8633a16c98639fd2f9482cfd72b1097e3aa3453c9c8cbbfbad2f88e95ec48" exitCode=0
Sep 10 00:19:35.753420 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:35.753105    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj" event={"ID":"b866f960-c15a-4242-ac05-e1bce0cc0c69","Type":"ContainerDied","Data":"11b8633a16c98639fd2f9482cfd72b1097e3aa3453c9c8cbbfbad2f88e95ec48"}
Sep 10 00:19:35.753420 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:35.753158    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj" event={"ID":"b866f960-c15a-4242-ac05-e1bce0cc0c69","Type":"ContainerStarted","Data":"a209ec197f8d7433ac221b617bdb2a653604cf6041017731f142cb17fe193adc"}
Sep 10 00:19:36.890869 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:36.890848    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"
Sep 10 00:19:36.989786 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:36.989757    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b866f960-c15a-4242-ac05-e1bce0cc0c69-maas-api-service-ca\") pod \"b866f960-c15a-4242-ac05-e1bce0cc0c69\" (UID: \"b866f960-c15a-4242-ac05-e1bce0cc0c69\") "
Sep 10 00:19:36.990098 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:36.990075    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b866f960-c15a-4242-ac05-e1bce0cc0c69-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b866f960-c15a-4242-ac05-e1bce0cc0c69" (UID: "b866f960-c15a-4242-ac05-e1bce0cc0c69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:19:37.091207 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:37.091146    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b866f960-c15a-4242-ac05-e1bce0cc0c69-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:37.765708 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:37.765682    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"
Sep 10 00:19:37.765900 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:37.765724    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj" event={"ID":"b866f960-c15a-4242-ac05-e1bce0cc0c69","Type":"ContainerDied","Data":"a209ec197f8d7433ac221b617bdb2a653604cf6041017731f142cb17fe193adc"}
Sep 10 00:19:37.765900 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:37.765771    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a209ec197f8d7433ac221b617bdb2a653604cf6041017731f142cb17fe193adc"
Sep 10 00:19:39.084759 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:39.084715    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"]
Sep 10 00:19:39.104128 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:39.104031    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"]
Sep 10 00:19:39.130460 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:39.130430    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"]
Sep 10 00:19:40.542136 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:40.542107    2556 scope.go:117] "RemoveContainer" containerID="fc28416498bc641da0deb7e1d96fdb94b9226e4de4edefa5031eb39ab8b9121f"
Sep 10 00:19:40.551225 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:40.551202    2556 scope.go:117] "RemoveContainer" containerID="267e6f543aa7d7a8257cd87328af61307f05bae6e1f81cca3f541b25c0dbc9a1"
Sep 10 00:19:49.778343 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:49.778311    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl"]
Sep 10 00:19:49.781512 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:49.781487    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc0040-29816655-mk8nl"]
Sep 10 00:19:49.805481 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:49.805449    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"]
Sep 10 00:19:49.805733 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:49.805696    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8" podUID="38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7" containerName="payload-processing" containerID="cri-o://ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269" gracePeriod=30
Sep 10 00:19:49.813746 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:49.813723    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"]
Sep 10 00:19:49.813943 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:49.813906    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27" podUID="7afd4e08-7c7d-4525-88b4-2aaa32c171b9" containerName="payload-pre-processing" containerID="cri-o://841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325" gracePeriod=30
Sep 10 00:19:49.970224 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:49.970028    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"]
Sep 10 00:19:49.973217 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:49.973186    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc0040-4sccj"]
Sep 10 00:19:50.098763 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.098743    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:19:50.101526 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.101508    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:19:50.203020 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.202991    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-plugins-config-volume\") pod \"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7\" (UID: \"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7\") "
Sep 10 00:19:50.203170 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.203027    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cr2z\" (UniqueName: \"kubernetes.io/projected/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-kube-api-access-8cr2z\") pod \"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7\" (UID: \"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7\") "
Sep 10 00:19:50.203170 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.203120    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4rj8\" (UniqueName: \"kubernetes.io/projected/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-kube-api-access-l4rj8\") pod \"7afd4e08-7c7d-4525-88b4-2aaa32c171b9\" (UID: \"7afd4e08-7c7d-4525-88b4-2aaa32c171b9\") "
Sep 10 00:19:50.203261 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.203170    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-plugins-config-volume\") pod \"7afd4e08-7c7d-4525-88b4-2aaa32c171b9\" (UID: \"7afd4e08-7c7d-4525-88b4-2aaa32c171b9\") "
Sep 10 00:19:50.203429 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.203405    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7" (UID: "38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:19:50.203610 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.203588    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7afd4e08-7c7d-4525-88b4-2aaa32c171b9" (UID: "7afd4e08-7c7d-4525-88b4-2aaa32c171b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:19:50.205151 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.205130    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-kube-api-access-l4rj8" (OuterVolumeSpecName: "kube-api-access-l4rj8") pod "7afd4e08-7c7d-4525-88b4-2aaa32c171b9" (UID: "7afd4e08-7c7d-4525-88b4-2aaa32c171b9"). InnerVolumeSpecName "kube-api-access-l4rj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:19:50.205224 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.205184    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-kube-api-access-8cr2z" (OuterVolumeSpecName: "kube-api-access-8cr2z") pod "38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7" (UID: "38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7"). InnerVolumeSpecName "kube-api-access-8cr2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:19:50.267873 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.267836    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b866f960-c15a-4242-ac05-e1bce0cc0c69" path="/var/lib/kubelet/pods/b866f960-c15a-4242-ac05-e1bce0cc0c69/volumes"
Sep 10 00:19:50.268252 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.268232    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fad82862-c1ab-45b0-8c30-a65bd667d340" path="/var/lib/kubelet/pods/fad82862-c1ab-45b0-8c30-a65bd667d340/volumes"
Sep 10 00:19:50.304318 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.304298    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:50.304414 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.304321    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cr2z\" (UniqueName: \"kubernetes.io/projected/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7-kube-api-access-8cr2z\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:50.304414 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.304337    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4rj8\" (UniqueName: \"kubernetes.io/projected/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-kube-api-access-l4rj8\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:50.304414 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.304351    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7afd4e08-7c7d-4525-88b4-2aaa32c171b9-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:50.846937 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.846900    2556 generic.go:356] "Generic (PLEG): container finished" podID="38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7" containerID="ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269" exitCode=0
Sep 10 00:19:50.847333 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.846945    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"
Sep 10 00:19:50.847333 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.846981    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8" event={"ID":"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7","Type":"ContainerDied","Data":"ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269"}
Sep 10 00:19:50.847333 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.847032    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8" event={"ID":"38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7","Type":"ContainerDied","Data":"b2183fef4170ff0b99f9e22bb7a70f6ba7e59474b72e07f32f0b27b2b2ff0c7b"}
Sep 10 00:19:50.847333 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.847066    2556 scope.go:117] "RemoveContainer" containerID="ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269"
Sep 10 00:19:50.848628 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.848603    2556 generic.go:356] "Generic (PLEG): container finished" podID="7afd4e08-7c7d-4525-88b4-2aaa32c171b9" containerID="841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325" exitCode=0
Sep 10 00:19:50.848751 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.848652    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"
Sep 10 00:19:50.848751 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.848693    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27" event={"ID":"7afd4e08-7c7d-4525-88b4-2aaa32c171b9","Type":"ContainerDied","Data":"841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325"}
Sep 10 00:19:50.848751 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.848742    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27" event={"ID":"7afd4e08-7c7d-4525-88b4-2aaa32c171b9","Type":"ContainerDied","Data":"36098ce1b46a764ec99ebab7e9cb00a37537e7eb16053bd0def9f80c8a607471"}
Sep 10 00:19:50.856450 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.856434    2556 scope.go:117] "RemoveContainer" containerID="ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269"
Sep 10 00:19:50.856723 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:19:50.856700    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269\": container with ID starting with ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269 not found: ID does not exist" containerID="ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269"
Sep 10 00:19:50.856815 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.856730    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269"} err="failed to get container status \"ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269\": rpc error: code = NotFound desc = could not find container \"ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269\": container with ID starting with ba9edd708d45e005a4e3b635a35d5f5ecd56cc2ca56ff8f08b5b8b2564a28269 not found: ID does not exist"
Sep 10 00:19:50.856815 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.856761    2556 scope.go:117] "RemoveContainer" containerID="841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325"
Sep 10 00:19:50.864900 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.864886    2556 scope.go:117] "RemoveContainer" containerID="841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325"
Sep 10 00:19:50.865261 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:19:50.865244    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325\": container with ID starting with 841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325 not found: ID does not exist" containerID="841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325"
Sep 10 00:19:50.865325 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.865269    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325"} err="failed to get container status \"841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325\": rpc error: code = NotFound desc = could not find container \"841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325\": container with ID starting with 841379509ee0c0dfd5b3aa5afe1609019d0b219597caf7aaf344abe5b12fb325 not found: ID does not exist"
Sep 10 00:19:50.865378 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.865322    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"]
Sep 10 00:19:50.868776 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.868758    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc0040-b7f7ddf8c-r4qw8"]
Sep 10 00:19:50.878759 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.878733    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"]
Sep 10 00:19:50.882199 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:50.882177    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-cc0040-97c4f67bb-rhh27"]
Sep 10 00:19:52.267921 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:52.267889    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7" path="/var/lib/kubelet/pods/38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7/volumes"
Sep 10 00:19:52.268310 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:52.268294    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7afd4e08-7c7d-4525-88b4-2aaa32c171b9" path="/var/lib/kubelet/pods/7afd4e08-7c7d-4525-88b4-2aaa32c171b9/volumes"
Sep 10 00:19:54.105326 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.105281    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf" podUID="95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" containerName="main" containerID="cri-o://67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4" gracePeriod=45
Sep 10 00:19:54.126428 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.126397    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn" podUID="223834c5-5b10-471b-9e03-7bbd3aa817a9" containerName="main" containerID="cri-o://9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161" gracePeriod=45
Sep 10 00:19:54.149288 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.149262    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh" podUID="35ac38ec-a740-4763-98b3-cbed9245cdfa" containerName="main" containerID="cri-o://cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e" gracePeriod=45
Sep 10 00:19:54.409149 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.409128    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:19:54.437236 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.437210    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-dshm\") pod \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") "
Sep 10 00:19:54.437374 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.437256    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khghd\" (UniqueName: \"kubernetes.io/projected/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-kube-api-access-khghd\") pod \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") "
Sep 10 00:19:54.437374 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.437302    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-home\") pod \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") "
Sep 10 00:19:54.437374 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.437332    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-model-cache\") pod \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") "
Sep 10 00:19:54.437790 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.437432    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tls-certs\") pod \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") "
Sep 10 00:19:54.437790 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.437458    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tmp-dir\") pod \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\" (UID: \"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391\") "
Sep 10 00:19:54.437790 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.437598    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-home" (OuterVolumeSpecName: "home") pod "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" (UID: "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.437790 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.437698    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-model-cache" (OuterVolumeSpecName: "model-cache") pod "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" (UID: "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.438022 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.437821    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.438022 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.437840    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.438022 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.437842    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" (UID: "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.440172 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.440140    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" (UID: "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:19:54.440172 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.440155    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-kube-api-access-khghd" (OuterVolumeSpecName: "kube-api-access-khghd") pod "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" (UID: "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391"). InnerVolumeSpecName "kube-api-access-khghd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:19:54.440351 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.440184    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-dshm" (OuterVolumeSpecName: "dshm") pod "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" (UID: "95bf0a43-fe6d-4412-8eeb-ecc40c4a0391"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.446236 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.446215    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:19:54.450763 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.450745    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:19:54.538818 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.538791    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223834c5-5b10-471b-9e03-7bbd3aa817a9-tls-certs\") pod \"223834c5-5b10-471b-9e03-7bbd3aa817a9\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") "
Sep 10 00:19:54.538974 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.538841    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ac38ec-a740-4763-98b3-cbed9245cdfa-tls-certs\") pod \"35ac38ec-a740-4763-98b3-cbed9245cdfa\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") "
Sep 10 00:19:54.538974 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.538866    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-home\") pod \"35ac38ec-a740-4763-98b3-cbed9245cdfa\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") "
Sep 10 00:19:54.538974 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.538882    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-model-cache\") pod \"35ac38ec-a740-4763-98b3-cbed9245cdfa\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") "
Sep 10 00:19:54.539170 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.538990    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-dshm\") pod \"223834c5-5b10-471b-9e03-7bbd3aa817a9\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") "
Sep 10 00:19:54.539170 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539069    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-tmp-dir\") pod \"223834c5-5b10-471b-9e03-7bbd3aa817a9\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") "
Sep 10 00:19:54.539170 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539099    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-model-cache" (OuterVolumeSpecName: "model-cache") pod "35ac38ec-a740-4763-98b3-cbed9245cdfa" (UID: "35ac38ec-a740-4763-98b3-cbed9245cdfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.539170 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539115    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-home\") pod \"223834c5-5b10-471b-9e03-7bbd3aa817a9\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") "
Sep 10 00:19:54.539170 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539164    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-tmp-dir\") pod \"35ac38ec-a740-4763-98b3-cbed9245cdfa\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") "
Sep 10 00:19:54.539450 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539183    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-home" (OuterVolumeSpecName: "home") pod "35ac38ec-a740-4763-98b3-cbed9245cdfa" (UID: "35ac38ec-a740-4763-98b3-cbed9245cdfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.539450 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539203    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbw8w\" (UniqueName: \"kubernetes.io/projected/223834c5-5b10-471b-9e03-7bbd3aa817a9-kube-api-access-nbw8w\") pod \"223834c5-5b10-471b-9e03-7bbd3aa817a9\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") "
Sep 10 00:19:54.539450 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539252    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdg6n\" (UniqueName: \"kubernetes.io/projected/35ac38ec-a740-4763-98b3-cbed9245cdfa-kube-api-access-hdg6n\") pod \"35ac38ec-a740-4763-98b3-cbed9245cdfa\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") "
Sep 10 00:19:54.539450 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539275    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-model-cache\") pod \"223834c5-5b10-471b-9e03-7bbd3aa817a9\" (UID: \"223834c5-5b10-471b-9e03-7bbd3aa817a9\") "
Sep 10 00:19:54.539450 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539302    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-dshm\") pod \"35ac38ec-a740-4763-98b3-cbed9245cdfa\" (UID: \"35ac38ec-a740-4763-98b3-cbed9245cdfa\") "
Sep 10 00:19:54.539450 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539325    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "223834c5-5b10-471b-9e03-7bbd3aa817a9" (UID: "223834c5-5b10-471b-9e03-7bbd3aa817a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.539450 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539337    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-home" (OuterVolumeSpecName: "home") pod "223834c5-5b10-471b-9e03-7bbd3aa817a9" (UID: "223834c5-5b10-471b-9e03-7bbd3aa817a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.539450 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539392    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35ac38ec-a740-4763-98b3-cbed9245cdfa" (UID: "35ac38ec-a740-4763-98b3-cbed9245cdfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.539872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539487    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "223834c5-5b10-471b-9e03-7bbd3aa817a9" (UID: "223834c5-5b10-471b-9e03-7bbd3aa817a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.539872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539684    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.539872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539707    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.539872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539720    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.539872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539732    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.539872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539746    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.539872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539757    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.539872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539769    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.539872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539783    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khghd\" (UniqueName: \"kubernetes.io/projected/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391-kube-api-access-khghd\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.539872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539795    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.539872 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.539808    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.541290 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.541267    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223834c5-5b10-471b-9e03-7bbd3aa817a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "223834c5-5b10-471b-9e03-7bbd3aa817a9" (UID: "223834c5-5b10-471b-9e03-7bbd3aa817a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:19:54.541486 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.541463    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-dshm" (OuterVolumeSpecName: "dshm") pod "223834c5-5b10-471b-9e03-7bbd3aa817a9" (UID: "223834c5-5b10-471b-9e03-7bbd3aa817a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.541574 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.541516    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35ac38ec-a740-4763-98b3-cbed9245cdfa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35ac38ec-a740-4763-98b3-cbed9245cdfa" (UID: "35ac38ec-a740-4763-98b3-cbed9245cdfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:19:54.541574 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.541561    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-dshm" (OuterVolumeSpecName: "dshm") pod "35ac38ec-a740-4763-98b3-cbed9245cdfa" (UID: "35ac38ec-a740-4763-98b3-cbed9245cdfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:19:54.541687 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.541671    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ac38ec-a740-4763-98b3-cbed9245cdfa-kube-api-access-hdg6n" (OuterVolumeSpecName: "kube-api-access-hdg6n") pod "35ac38ec-a740-4763-98b3-cbed9245cdfa" (UID: "35ac38ec-a740-4763-98b3-cbed9245cdfa"). InnerVolumeSpecName "kube-api-access-hdg6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:19:54.541746 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.541710    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223834c5-5b10-471b-9e03-7bbd3aa817a9-kube-api-access-nbw8w" (OuterVolumeSpecName: "kube-api-access-nbw8w") pod "223834c5-5b10-471b-9e03-7bbd3aa817a9" (UID: "223834c5-5b10-471b-9e03-7bbd3aa817a9"). InnerVolumeSpecName "kube-api-access-nbw8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:19:54.641094 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.641026    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223834c5-5b10-471b-9e03-7bbd3aa817a9-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.641094 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.641079    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbw8w\" (UniqueName: \"kubernetes.io/projected/223834c5-5b10-471b-9e03-7bbd3aa817a9-kube-api-access-nbw8w\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.641094 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.641090    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdg6n\" (UniqueName: \"kubernetes.io/projected/35ac38ec-a740-4763-98b3-cbed9245cdfa-kube-api-access-hdg6n\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.641094 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.641099    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35ac38ec-a740-4763-98b3-cbed9245cdfa-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.641283 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.641108    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223834c5-5b10-471b-9e03-7bbd3aa817a9-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.641283 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.641116    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35ac38ec-a740-4763-98b3-cbed9245cdfa-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:19:54.874192 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.874164    2556 generic.go:356] "Generic (PLEG): container finished" podID="35ac38ec-a740-4763-98b3-cbed9245cdfa" containerID="cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e" exitCode=0
Sep 10 00:19:54.874319 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.874211    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"
Sep 10 00:19:54.874319 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.874252    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh" event={"ID":"35ac38ec-a740-4763-98b3-cbed9245cdfa","Type":"ContainerDied","Data":"cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e"}
Sep 10 00:19:54.874319 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.874287    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh" event={"ID":"35ac38ec-a740-4763-98b3-cbed9245cdfa","Type":"ContainerDied","Data":"82a619b78fd9074f040b678d657c0e093d1f1a18632c2ce6da4bb24ba8875825"}
Sep 10 00:19:54.874319 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.874305    2556 scope.go:117] "RemoveContainer" containerID="cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e"
Sep 10 00:19:54.875761 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.875743    2556 generic.go:356] "Generic (PLEG): container finished" podID="223834c5-5b10-471b-9e03-7bbd3aa817a9" containerID="9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161" exitCode=0
Sep 10 00:19:54.875867 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.875773    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn" event={"ID":"223834c5-5b10-471b-9e03-7bbd3aa817a9","Type":"ContainerDied","Data":"9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161"}
Sep 10 00:19:54.875867 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.875799    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"
Sep 10 00:19:54.875867 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.875807    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn" event={"ID":"223834c5-5b10-471b-9e03-7bbd3aa817a9","Type":"ContainerDied","Data":"d4fc43ec7790d0bf6478dd6d3f6eac899f370813f89932e5aa6e63d416ca6937"}
Sep 10 00:19:54.877240 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.877202    2556 generic.go:356] "Generic (PLEG): container finished" podID="95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" containerID="67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4" exitCode=0
Sep 10 00:19:54.877240 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.877229    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf" event={"ID":"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391","Type":"ContainerDied","Data":"67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4"}
Sep 10 00:19:54.877383 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.877258    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"
Sep 10 00:19:54.877383 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.877260    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf" event={"ID":"95bf0a43-fe6d-4412-8eeb-ecc40c4a0391","Type":"ContainerDied","Data":"4f83256380eef72a49a1f9b3f75e0bca6491c0c33451497f0c80bccc949c3dbe"}
Sep 10 00:19:54.884034 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.884013    2556 scope.go:117] "RemoveContainer" containerID="cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e"
Sep 10 00:19:54.884350 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:19:54.884331    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e\": container with ID starting with cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e not found: ID does not exist" containerID="cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e"
Sep 10 00:19:54.884422 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.884357    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e"} err="failed to get container status \"cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e\": rpc error: code = NotFound desc = could not find container \"cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e\": container with ID starting with cd044ea236cb35d6ddd8f7cac31b5bfad5a891d4bdd29b17e1388652f1b8968e not found: ID does not exist"
Sep 10 00:19:54.884422 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.884375    2556 scope.go:117] "RemoveContainer" containerID="9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161"
Sep 10 00:19:54.893223 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.893171    2556 scope.go:117] "RemoveContainer" containerID="9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161"
Sep 10 00:19:54.893441 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:19:54.893423    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161\": container with ID starting with 9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161 not found: ID does not exist" containerID="9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161"
Sep 10 00:19:54.893493 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.893447    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161"} err="failed to get container status \"9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161\": rpc error: code = NotFound desc = could not find container \"9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161\": container with ID starting with 9a028950a9bcbc6668a603990e18debf219f7de6f5af6a26631a630c98e55161 not found: ID does not exist"
Sep 10 00:19:54.893493 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.893462    2556 scope.go:117] "RemoveContainer" containerID="67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4"
Sep 10 00:19:54.901443 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.901426    2556 scope.go:117] "RemoveContainer" containerID="67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4"
Sep 10 00:19:54.901677 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:19:54.901662    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4\": container with ID starting with 67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4 not found: ID does not exist" containerID="67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4"
Sep 10 00:19:54.901727 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.901682    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4"} err="failed to get container status \"67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4\": rpc error: code = NotFound desc = could not find container \"67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4\": container with ID starting with 67e6b296776cc5a3099b624a604656d6b0358547e578408640186091a678e1b4 not found: ID does not exist"
Sep 10 00:19:54.903303 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.903284    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"]
Sep 10 00:19:54.905155 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.905136    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/unconfigured-w4-cc0040-kserve-6b8645956b-gxhzh"]
Sep 10 00:19:54.914484 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.914465    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"]
Sep 10 00:19:54.918145 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.918121    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/distinct-2-w4-cc0040-kserve-586fcd64d6-x7kxf"]
Sep 10 00:19:54.927642 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.927624    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"]
Sep 10 00:19:54.931125 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:54.931106    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cc0040/embedding-w4-cc0040-kserve-5fb8fbddc8-ngvvn"]
Sep 10 00:19:56.268859 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:56.268826    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223834c5-5b10-471b-9e03-7bbd3aa817a9" path="/var/lib/kubelet/pods/223834c5-5b10-471b-9e03-7bbd3aa817a9/volumes"
Sep 10 00:19:56.269273 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:56.269259    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35ac38ec-a740-4763-98b3-cbed9245cdfa" path="/var/lib/kubelet/pods/35ac38ec-a740-4763-98b3-cbed9245cdfa/volumes"
Sep 10 00:19:56.269624 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:19:56.269610    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" path="/var/lib/kubelet/pods/95bf0a43-fe6d-4412-8eeb-ecc40c4a0391/volumes"
Sep 10 00:20:00.807597 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.807564    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7792-9e6326"]
Sep 10 00:20:00.807989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.807944    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" containerName="main"
Sep 10 00:20:00.807989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.807956    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" containerName="main"
Sep 10 00:20:00.807989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.807974    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b866f960-c15a-4242-ac05-e1bce0cc0c69" containerName="revoke-keys"
Sep 10 00:20:00.807989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.807980    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b866f960-c15a-4242-ac05-e1bce0cc0c69" containerName="revoke-keys"
Sep 10 00:20:00.807989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.807990    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223834c5-5b10-471b-9e03-7bbd3aa817a9" containerName="main"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.807995    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="223834c5-5b10-471b-9e03-7bbd3aa817a9" containerName="main"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808006    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7" containerName="payload-processing"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808011    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7" containerName="payload-processing"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808019    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35ac38ec-a740-4763-98b3-cbed9245cdfa" containerName="main"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808024    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="35ac38ec-a740-4763-98b3-cbed9245cdfa" containerName="main"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808035    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7afd4e08-7c7d-4525-88b4-2aaa32c171b9" containerName="payload-pre-processing"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808041    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7afd4e08-7c7d-4525-88b4-2aaa32c171b9" containerName="payload-pre-processing"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808103    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="223834c5-5b10-471b-9e03-7bbd3aa817a9" containerName="main"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808112    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f14d8b-d9e0-4c21-8638-4cab4e6ad9d7" containerName="payload-processing"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808119    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7afd4e08-7c7d-4525-88b4-2aaa32c171b9" containerName="payload-pre-processing"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808124    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="35ac38ec-a740-4763-98b3-cbed9245cdfa" containerName="main"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808132    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="95bf0a43-fe6d-4412-8eeb-ecc40c4a0391" containerName="main"
Sep 10 00:20:00.808177 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.808137    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b866f960-c15a-4242-ac05-e1bce0cc0c69" containerName="revoke-keys"
Sep 10 00:20:00.812344 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.812322    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-9e6326"
Sep 10 00:20:00.814325 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.814301    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7792-9e6326"]
Sep 10 00:20:00.814458 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.814439    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gtwx5\""
Sep 10 00:20:00.894700 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.894676    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gpxm\" (UniqueName: \"kubernetes.io/projected/c59baf69-bd48-49b3-a1d6-d5d0966c8240-kube-api-access-5gpxm\") pod \"test-curl-7792-9e6326\" (UID: \"c59baf69-bd48-49b3-a1d6-d5d0966c8240\") " pod="opendatahub/test-curl-7792-9e6326"
Sep 10 00:20:00.995544 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:00.995521    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpxm\" (UniqueName: \"kubernetes.io/projected/c59baf69-bd48-49b3-a1d6-d5d0966c8240-kube-api-access-5gpxm\") pod \"test-curl-7792-9e6326\" (UID: \"c59baf69-bd48-49b3-a1d6-d5d0966c8240\") " pod="opendatahub/test-curl-7792-9e6326"
Sep 10 00:20:01.002611 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:01.002591    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gpxm\" (UniqueName: \"kubernetes.io/projected/c59baf69-bd48-49b3-a1d6-d5d0966c8240-kube-api-access-5gpxm\") pod \"test-curl-7792-9e6326\" (UID: \"c59baf69-bd48-49b3-a1d6-d5d0966c8240\") " pod="opendatahub/test-curl-7792-9e6326"
Sep 10 00:20:01.122074 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:01.121990    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-9e6326"
Sep 10 00:20:01.388401 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:01.388377    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7792-9e6326"]
Sep 10 00:20:01.390262 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:20:01.390234    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc59baf69_bd48_49b3_a1d6_d5d0966c8240.slice/crio-3e16309de872b98606e56dffeca73fb1f108499c69632f0a8ad42503031ab37e WatchSource:0}: Error finding container 3e16309de872b98606e56dffeca73fb1f108499c69632f0a8ad42503031ab37e: Status 404 returned error can't find the container with id 3e16309de872b98606e56dffeca73fb1f108499c69632f0a8ad42503031ab37e
Sep 10 00:20:01.921542 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:01.921501    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7792-9e6326" event={"ID":"c59baf69-bd48-49b3-a1d6-d5d0966c8240","Type":"ContainerStarted","Data":"e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16"}
Sep 10 00:20:01.921542 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:01.921545    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7792-9e6326" event={"ID":"c59baf69-bd48-49b3-a1d6-d5d0966c8240","Type":"ContainerStarted","Data":"3e16309de872b98606e56dffeca73fb1f108499c69632f0a8ad42503031ab37e"}
Sep 10 00:20:01.934071 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:01.934005    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7792-9e6326" podStartSLOduration=1.93399112 podStartE2EDuration="1.93399112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:20:01.933375892 +0000 UTC m=+1882.184511936" watchObservedRunningTime="2026-09-10 00:20:01.93399112 +0000 UTC m=+1882.185127163"
Sep 10 00:20:02.527951 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:02.527903    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7792-9e6326"]
Sep 10 00:20:02.532008 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:02.531964    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7792-9e6326"]
Sep 10 00:20:02.926953 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:20:02.926917    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7792-9e6326" container="test-curl-7792-9e6326"
Sep 10 00:20:03.932734 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:03.932685    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7792-9e6326" podUID="c59baf69-bd48-49b3-a1d6-d5d0966c8240" containerName="test-curl-7792-9e6326" containerID="cri-o://e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16" gracePeriod=2
Sep 10 00:20:03.934823 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:20:03.934789    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7792-9e6326\" is forbidden: User \"system:node:ip-10-0-139-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-211.ec2.internal' and this object" podUID="c59baf69-bd48-49b3-a1d6-d5d0966c8240" pod="opendatahub/test-curl-7792-9e6326"
Sep 10 00:20:06.112779 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:06.112757    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-9e6326"
Sep 10 00:20:06.139687 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:06.139665    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gpxm\" (UniqueName: \"kubernetes.io/projected/c59baf69-bd48-49b3-a1d6-d5d0966c8240-kube-api-access-5gpxm\") pod \"c59baf69-bd48-49b3-a1d6-d5d0966c8240\" (UID: \"c59baf69-bd48-49b3-a1d6-d5d0966c8240\") "
Sep 10 00:20:06.141487 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:06.141463    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59baf69-bd48-49b3-a1d6-d5d0966c8240-kube-api-access-5gpxm" (OuterVolumeSpecName: "kube-api-access-5gpxm") pod "c59baf69-bd48-49b3-a1d6-d5d0966c8240" (UID: "c59baf69-bd48-49b3-a1d6-d5d0966c8240"). InnerVolumeSpecName "kube-api-access-5gpxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:20:06.240896 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:06.240873    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gpxm\" (UniqueName: \"kubernetes.io/projected/c59baf69-bd48-49b3-a1d6-d5d0966c8240-kube-api-access-5gpxm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:20:06.268280 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:06.268254    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c59baf69-bd48-49b3-a1d6-d5d0966c8240" path="/var/lib/kubelet/pods/c59baf69-bd48-49b3-a1d6-d5d0966c8240/volumes"
Sep 10 00:20:06.950696 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:06.950662    2556 generic.go:356] "Generic (PLEG): container finished" podID="c59baf69-bd48-49b3-a1d6-d5d0966c8240" containerID="e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16" exitCode=137
Sep 10 00:20:06.950876 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:06.950713    2556 scope.go:117] "RemoveContainer" containerID="e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16"
Sep 10 00:20:06.950876 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:06.950724    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-9e6326"
Sep 10 00:20:06.959593 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:06.959569    2556 scope.go:117] "RemoveContainer" containerID="e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16"
Sep 10 00:20:06.959910 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:20:06.959890    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16\": container with ID starting with e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16 not found: ID does not exist" containerID="e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16"
Sep 10 00:20:06.959958 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:06.959917    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16"} err="failed to get container status \"e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16\": rpc error: code = NotFound desc = could not find container \"e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16\": container with ID starting with e45980e3e401f4147767d06683e64d2c12e11c432923348c2f17e0825dcc6f16 not found: ID does not exist"
Sep 10 00:20:15.154642 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.154609    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"]
Sep 10 00:20:15.155024 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.155012    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59baf69-bd48-49b3-a1d6-d5d0966c8240" containerName="test-curl-7792-9e6326"
Sep 10 00:20:15.155024 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.155024    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59baf69-bd48-49b3-a1d6-d5d0966c8240" containerName="test-curl-7792-9e6326"
Sep 10 00:20:15.155237 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.155111    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59baf69-bd48-49b3-a1d6-d5d0966c8240" containerName="test-curl-7792-9e6326"
Sep 10 00:20:15.158023 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.158004    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.161302 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.161282    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"default-dockercfg-bkdjc\""
Sep 10 00:20:15.161404 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.161349    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"sub-test-model-1de9a7-kserve-self-signed-certs\""
Sep 10 00:20:15.161404 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.161369    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"kube-root-ca.crt\""
Sep 10 00:20:15.161404 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.161358    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1de9a7\"/\"openshift-service-ca.crt\""
Sep 10 00:20:15.165567 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.165546    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"]
Sep 10 00:20:15.213768 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.213743    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-model-cache\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.213888 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.213850    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-home\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.213963 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.213901    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28c8979-95ed-4d14-aa58-0a98a4a77997-tls-certs\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.213963 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.213954    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-tmp-dir\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.214049 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.214007    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rzj8\" (UniqueName: \"kubernetes.io/projected/b28c8979-95ed-4d14-aa58-0a98a4a77997-kube-api-access-8rzj8\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.214049 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.214035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-dshm\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.314767 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.314738    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-dshm\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.314911 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.314776    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-model-cache\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.314911 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.314834    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-home\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.314911 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.314857    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28c8979-95ed-4d14-aa58-0a98a4a77997-tls-certs\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.314911 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.314875    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-tmp-dir\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.315183 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.314922    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rzj8\" (UniqueName: \"kubernetes.io/projected/b28c8979-95ed-4d14-aa58-0a98a4a77997-kube-api-access-8rzj8\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.315292 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.315274    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-home\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.315349 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.315321    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-model-cache\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.315349 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.315334    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-tmp-dir\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.317107 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.317089    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-dshm\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.317270 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.317255    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28c8979-95ed-4d14-aa58-0a98a4a77997-tls-certs\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.323721 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.323704    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rzj8\" (UniqueName: \"kubernetes.io/projected/b28c8979-95ed-4d14-aa58-0a98a4a77997-kube-api-access-8rzj8\") pod \"sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") " pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.468261 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.468235    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:15.738418 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:15.738390    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"]
Sep 10 00:20:15.740904 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:20:15.740875    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb28c8979_95ed_4d14_aa58_0a98a4a77997.slice/crio-479aca9c3080258e794c7d5bfc9cedff0d861651beab5fa0a0b14633f02c01f8 WatchSource:0}: Error finding container 479aca9c3080258e794c7d5bfc9cedff0d861651beab5fa0a0b14633f02c01f8: Status 404 returned error can't find the container with id 479aca9c3080258e794c7d5bfc9cedff0d861651beab5fa0a0b14633f02c01f8
Sep 10 00:20:16.000782 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:16.000704    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj" event={"ID":"b28c8979-95ed-4d14-aa58-0a98a4a77997","Type":"ContainerStarted","Data":"948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0"}
Sep 10 00:20:16.000782 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:16.000748    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj" event={"ID":"b28c8979-95ed-4d14-aa58-0a98a4a77997","Type":"ContainerStarted","Data":"479aca9c3080258e794c7d5bfc9cedff0d861651beab5fa0a0b14633f02c01f8"}
Sep 10 00:20:16.021609 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:16.021561    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj" podStartSLOduration=1.021546636 podStartE2EDuration="1.021546636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:20:16.020071531 +0000 UTC m=+1896.271207573" watchObservedRunningTime="2026-09-10 00:20:16.021546636 +0000 UTC m=+1896.272682713"
Sep 10 00:20:25.468503 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:25.468469    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:25.468966 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:25.468516    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:25.481173 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:25.481146    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:26.067755 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:26.067723    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:20:33.856787 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.856750    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"]
Sep 10 00:20:33.860688 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.860665    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:33.864256 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.864232    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"kube-root-ca.crt\""
Sep 10 00:20:33.864410 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.864311    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"openshift-service-ca.crt\""
Sep 10 00:20:33.864410 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.864325    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"sub-test-model-b1baac-kserve-self-signed-certs\""
Sep 10 00:20:33.864410 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.864333    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"default-dockercfg-jxkgn\""
Sep 10 00:20:33.870678 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.870656    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"]
Sep 10 00:20:33.995214 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.995177    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2797\" (UniqueName: \"kubernetes.io/projected/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-kube-api-access-z2797\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:33.995371 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.995224    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tls-certs\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:33.995371 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.995318    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-dshm\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:33.995371 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.995352    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tmp-dir\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:33.995508 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.995388    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-home\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:33.995508 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:33.995448    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-model-cache\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.096279 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.096252    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tls-certs\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.096417 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.096304    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-dshm\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.096417 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.096325    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tmp-dir\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.096417 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.096355    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-home\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.096417 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.096388    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-model-cache\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.096610 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.096474    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2797\" (UniqueName: \"kubernetes.io/projected/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-kube-api-access-z2797\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.096729 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.096708    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-home\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.096808 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.096755    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tmp-dir\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.096909 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.096889    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-model-cache\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.098677 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.098655    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-dshm\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.099093 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.099074    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tls-certs\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.104039 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.104013    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2797\" (UniqueName: \"kubernetes.io/projected/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-kube-api-access-z2797\") pod \"sub-test-model-b1baac-kserve-696c449b59-bzppl\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.171805 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.171750    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:34.441483 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:34.441460    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"]
Sep 10 00:20:34.443495 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:20:34.443468    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0d41660_dbf7_4e2a_8c12_ab3588e8cb10.slice/crio-800f0f41199674424faf76ad4d11c3adff028c12af5c01554f1e242c619fc818 WatchSource:0}: Error finding container 800f0f41199674424faf76ad4d11c3adff028c12af5c01554f1e242c619fc818: Status 404 returned error can't find the container with id 800f0f41199674424faf76ad4d11c3adff028c12af5c01554f1e242c619fc818
Sep 10 00:20:35.108948 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:35.108903    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl" event={"ID":"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10","Type":"ContainerStarted","Data":"ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d"}
Sep 10 00:20:35.109359 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:35.108956    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl" event={"ID":"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10","Type":"ContainerStarted","Data":"800f0f41199674424faf76ad4d11c3adff028c12af5c01554f1e242c619fc818"}
Sep 10 00:20:35.124899 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:35.124844    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl" podStartSLOduration=2.1248257170000002 podStartE2EDuration="2.124825717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:20:35.124118896 +0000 UTC m=+1915.375254938" watchObservedRunningTime="2026-09-10 00:20:35.124825717 +0000 UTC m=+1915.375961761"
Sep 10 00:20:44.172783 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:44.172750    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:44.173225 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:44.173017    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:44.185537 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:44.185516    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:20:45.178419 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:20:45.178388    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:21:21.076397 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:21.076362    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7792-c2e872"]
Sep 10 00:21:21.079829 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:21.079813    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-c2e872"
Sep 10 00:21:21.082136 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:21.082108    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gtwx5\""
Sep 10 00:21:21.083469 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:21.083446    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7792-c2e872"]
Sep 10 00:21:21.131530 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:21.131508    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zl88\" (UniqueName: \"kubernetes.io/projected/8484a80b-5a06-40b5-bd3c-640b6da1dd84-kube-api-access-6zl88\") pod \"test-curl-7792-c2e872\" (UID: \"8484a80b-5a06-40b5-bd3c-640b6da1dd84\") " pod="opendatahub/test-curl-7792-c2e872"
Sep 10 00:21:21.231895 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:21.231864    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zl88\" (UniqueName: \"kubernetes.io/projected/8484a80b-5a06-40b5-bd3c-640b6da1dd84-kube-api-access-6zl88\") pod \"test-curl-7792-c2e872\" (UID: \"8484a80b-5a06-40b5-bd3c-640b6da1dd84\") " pod="opendatahub/test-curl-7792-c2e872"
Sep 10 00:21:21.239797 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:21.239769    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zl88\" (UniqueName: \"kubernetes.io/projected/8484a80b-5a06-40b5-bd3c-640b6da1dd84-kube-api-access-6zl88\") pod \"test-curl-7792-c2e872\" (UID: \"8484a80b-5a06-40b5-bd3c-640b6da1dd84\") " pod="opendatahub/test-curl-7792-c2e872"
Sep 10 00:21:21.390395 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:21.390332    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-c2e872"
Sep 10 00:21:21.662238 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:21.662210    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7792-c2e872"]
Sep 10 00:21:21.663970 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:21:21.663943    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8484a80b_5a06_40b5_bd3c_640b6da1dd84.slice/crio-dbacaa7a30ed81cf61d8f3cf709e5a254f4b4a149f9bdbcc1c93e1e178ab5e7b WatchSource:0}: Error finding container dbacaa7a30ed81cf61d8f3cf709e5a254f4b4a149f9bdbcc1c93e1e178ab5e7b: Status 404 returned error can't find the container with id dbacaa7a30ed81cf61d8f3cf709e5a254f4b4a149f9bdbcc1c93e1e178ab5e7b
Sep 10 00:21:22.375614 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:22.375576    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7792-c2e872" event={"ID":"8484a80b-5a06-40b5-bd3c-640b6da1dd84","Type":"ContainerStarted","Data":"9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0"}
Sep 10 00:21:22.375614 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:22.375616    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7792-c2e872" event={"ID":"8484a80b-5a06-40b5-bd3c-640b6da1dd84","Type":"ContainerStarted","Data":"dbacaa7a30ed81cf61d8f3cf709e5a254f4b4a149f9bdbcc1c93e1e178ab5e7b"}
Sep 10 00:21:22.393047 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:22.392996    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7792-c2e872" podStartSLOduration=1.392980107 podStartE2EDuration="1.392980107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:21:22.391874048 +0000 UTC m=+1962.643010094" watchObservedRunningTime="2026-09-10 00:21:22.392980107 +0000 UTC m=+1962.644116150"
Sep 10 00:21:22.995011 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:22.994974    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7792-c2e872"]
Sep 10 00:21:22.997467 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:22.997428    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7792-c2e872"]
Sep 10 00:21:23.381177 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:21:23.381137    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7792-c2e872" container="test-curl-7792-c2e872"
Sep 10 00:21:24.386753 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:24.386692    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7792-c2e872" podUID="8484a80b-5a06-40b5-bd3c-640b6da1dd84" containerName="test-curl-7792-c2e872" containerID="cri-o://9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0" gracePeriod=2
Sep 10 00:21:25.036947 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:25.036916    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"]
Sep 10 00:21:25.841358 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:25.841325    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"]
Sep 10 00:21:26.554980 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:26.554955    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-c2e872"
Sep 10 00:21:26.576310 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:26.576288    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zl88\" (UniqueName: \"kubernetes.io/projected/8484a80b-5a06-40b5-bd3c-640b6da1dd84-kube-api-access-6zl88\") pod \"8484a80b-5a06-40b5-bd3c-640b6da1dd84\" (UID: \"8484a80b-5a06-40b5-bd3c-640b6da1dd84\") "
Sep 10 00:21:26.578173 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:26.578154    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8484a80b-5a06-40b5-bd3c-640b6da1dd84-kube-api-access-6zl88" (OuterVolumeSpecName: "kube-api-access-6zl88") pod "8484a80b-5a06-40b5-bd3c-640b6da1dd84" (UID: "8484a80b-5a06-40b5-bd3c-640b6da1dd84"). InnerVolumeSpecName "kube-api-access-6zl88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:21:26.677678 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:26.677626    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zl88\" (UniqueName: \"kubernetes.io/projected/8484a80b-5a06-40b5-bd3c-640b6da1dd84-kube-api-access-6zl88\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:27.405515 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:27.405486    2556 generic.go:356] "Generic (PLEG): container finished" podID="8484a80b-5a06-40b5-bd3c-640b6da1dd84" containerID="9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0" exitCode=137
Sep 10 00:21:27.405912 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:27.405525    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7792-c2e872"
Sep 10 00:21:27.405912 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:27.405585    2556 scope.go:117] "RemoveContainer" containerID="9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0"
Sep 10 00:21:27.414924 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:27.414907    2556 scope.go:117] "RemoveContainer" containerID="9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0"
Sep 10 00:21:27.415198 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:21:27.415175    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0\": container with ID starting with 9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0 not found: ID does not exist" containerID="9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0"
Sep 10 00:21:27.415257 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:27.415210    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0"} err="failed to get container status \"9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0\": rpc error: code = NotFound desc = could not find container \"9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0\": container with ID starting with 9d7e95f4f2656a5f6a64aedf75b10f9cd552b3910b63932f273de6aab1e043c0 not found: ID does not exist"
Sep 10 00:21:27.415779 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:21:27.415755    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7792-c2e872\" is forbidden: User \"system:node:ip-10-0-139-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-211.ec2.internal' and this object" podUID="8484a80b-5a06-40b5-bd3c-640b6da1dd84" pod="opendatahub/test-curl-7792-c2e872"
Sep 10 00:21:28.268286 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:28.268249    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8484a80b-5a06-40b5-bd3c-640b6da1dd84" path="/var/lib/kubelet/pods/8484a80b-5a06-40b5-bd3c-640b6da1dd84/volumes"
Sep 10 00:21:40.057881 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.057822    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj" podUID="b28c8979-95ed-4d14-aa58-0a98a4a77997" containerName="main" containerID="cri-o://948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0" gracePeriod=45
Sep 10 00:21:40.311438 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.311383    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:21:40.393046 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393019    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rzj8\" (UniqueName: \"kubernetes.io/projected/b28c8979-95ed-4d14-aa58-0a98a4a77997-kube-api-access-8rzj8\") pod \"b28c8979-95ed-4d14-aa58-0a98a4a77997\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") "
Sep 10 00:21:40.393196 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393092    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28c8979-95ed-4d14-aa58-0a98a4a77997-tls-certs\") pod \"b28c8979-95ed-4d14-aa58-0a98a4a77997\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") "
Sep 10 00:21:40.393196 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393117    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-model-cache\") pod \"b28c8979-95ed-4d14-aa58-0a98a4a77997\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") "
Sep 10 00:21:40.393196 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393172    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-tmp-dir\") pod \"b28c8979-95ed-4d14-aa58-0a98a4a77997\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") "
Sep 10 00:21:40.393371 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393273    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-home\") pod \"b28c8979-95ed-4d14-aa58-0a98a4a77997\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") "
Sep 10 00:21:40.393371 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393302    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-dshm\") pod \"b28c8979-95ed-4d14-aa58-0a98a4a77997\" (UID: \"b28c8979-95ed-4d14-aa58-0a98a4a77997\") "
Sep 10 00:21:40.393488 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393413    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-model-cache" (OuterVolumeSpecName: "model-cache") pod "b28c8979-95ed-4d14-aa58-0a98a4a77997" (UID: "b28c8979-95ed-4d14-aa58-0a98a4a77997"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:21:40.393488 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393420    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b28c8979-95ed-4d14-aa58-0a98a4a77997" (UID: "b28c8979-95ed-4d14-aa58-0a98a4a77997"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:21:40.393573 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393526    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-home" (OuterVolumeSpecName: "home") pod "b28c8979-95ed-4d14-aa58-0a98a4a77997" (UID: "b28c8979-95ed-4d14-aa58-0a98a4a77997"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:21:40.393644 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393627    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:40.393687 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393649    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:40.393687 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.393662    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:40.395249 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.395227    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28c8979-95ed-4d14-aa58-0a98a4a77997-kube-api-access-8rzj8" (OuterVolumeSpecName: "kube-api-access-8rzj8") pod "b28c8979-95ed-4d14-aa58-0a98a4a77997" (UID: "b28c8979-95ed-4d14-aa58-0a98a4a77997"). InnerVolumeSpecName "kube-api-access-8rzj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:21:40.395328 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.395253    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28c8979-95ed-4d14-aa58-0a98a4a77997-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b28c8979-95ed-4d14-aa58-0a98a4a77997" (UID: "b28c8979-95ed-4d14-aa58-0a98a4a77997"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:21:40.395328 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.395310    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-dshm" (OuterVolumeSpecName: "dshm") pod "b28c8979-95ed-4d14-aa58-0a98a4a77997" (UID: "b28c8979-95ed-4d14-aa58-0a98a4a77997"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:21:40.481468 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.481436    2556 generic.go:356] "Generic (PLEG): container finished" podID="b28c8979-95ed-4d14-aa58-0a98a4a77997" containerID="948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0" exitCode=0
Sep 10 00:21:40.481570 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.481484    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"
Sep 10 00:21:40.481570 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.481531    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj" event={"ID":"b28c8979-95ed-4d14-aa58-0a98a4a77997","Type":"ContainerDied","Data":"948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0"}
Sep 10 00:21:40.481643 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.481581    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj" event={"ID":"b28c8979-95ed-4d14-aa58-0a98a4a77997","Type":"ContainerDied","Data":"479aca9c3080258e794c7d5bfc9cedff0d861651beab5fa0a0b14633f02c01f8"}
Sep 10 00:21:40.481643 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.481602    2556 scope.go:117] "RemoveContainer" containerID="948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0"
Sep 10 00:21:40.494752 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.494723    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b28c8979-95ed-4d14-aa58-0a98a4a77997-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:40.494883 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.494755    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b28c8979-95ed-4d14-aa58-0a98a4a77997-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:40.494883 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.494769    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rzj8\" (UniqueName: \"kubernetes.io/projected/b28c8979-95ed-4d14-aa58-0a98a4a77997-kube-api-access-8rzj8\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:40.498511 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.498490    2556 scope.go:117] "RemoveContainer" containerID="948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0"
Sep 10 00:21:40.498768 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:21:40.498749    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0\": container with ID starting with 948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0 not found: ID does not exist" containerID="948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0"
Sep 10 00:21:40.498823 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.498776    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0"} err="failed to get container status \"948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0\": rpc error: code = NotFound desc = could not find container \"948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0\": container with ID starting with 948aff1e72f3579e21aeab4d642fdb2dcef13773295fc18d0cb667492e877fc0 not found: ID does not exist"
Sep 10 00:21:40.507512 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.507488    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"]
Sep 10 00:21:40.510644 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.510625    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1de9a7/sub-test-model-1de9a7-kserve-7d698b7df7-nnjxj"]
Sep 10 00:21:40.633290 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.633238    2556 scope.go:117] "RemoveContainer" containerID="688db57725beda236fb62fdcd897c9f6f9fe9e236aa6a5ebb5be6241451d99b1"
Sep 10 00:21:40.641147 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.641126    2556 scope.go:117] "RemoveContainer" containerID="0730d3831444e9a00c6330bf30e2f2667db415ae33ac6e5ae711444e3c8493ac"
Sep 10 00:21:40.648842 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.648819    2556 scope.go:117] "RemoveContainer" containerID="8bf82cca5ee9d341f027512c12b28039fa3d04a34089f500ee922ace97d6e0a2"
Sep 10 00:21:40.860912 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:40.860878    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl" podUID="b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" containerName="main" containerID="cri-o://ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d" gracePeriod=45
Sep 10 00:21:41.109599 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.109576    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:21:41.203049 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.203017    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2797\" (UniqueName: \"kubernetes.io/projected/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-kube-api-access-z2797\") pod \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") "
Sep 10 00:21:41.203216 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.203082    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tmp-dir\") pod \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") "
Sep 10 00:21:41.203216 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.203130    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-dshm\") pod \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") "
Sep 10 00:21:41.203216 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.203151    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tls-certs\") pod \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") "
Sep 10 00:21:41.203216 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.203195    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-model-cache\") pod \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") "
Sep 10 00:21:41.203423 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.203239    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-home\") pod \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\" (UID: \"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10\") "
Sep 10 00:21:41.203423 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.203317    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" (UID: "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:21:41.203530 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.203460    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" (UID: "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:21:41.203582 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.203526    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-home" (OuterVolumeSpecName: "home") pod "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" (UID: "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:21:41.203582 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.203552    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:41.203582 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.203572    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:41.205346 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.205319    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-dshm" (OuterVolumeSpecName: "dshm") pod "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" (UID: "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:21:41.205466 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.205351    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" (UID: "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:21:41.205528 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.205477    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-kube-api-access-z2797" (OuterVolumeSpecName: "kube-api-access-z2797") pod "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" (UID: "b0d41660-dbf7-4e2a-8c12-ab3588e8cb10"). InnerVolumeSpecName "kube-api-access-z2797". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:21:41.304993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.304916    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:41.304993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.304943    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2797\" (UniqueName: \"kubernetes.io/projected/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-kube-api-access-z2797\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:41.304993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.304954    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:41.304993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.304963    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:21:41.488699 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.488664    2556 generic.go:356] "Generic (PLEG): container finished" podID="b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" containerID="ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d" exitCode=0
Sep 10 00:21:41.488891 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.488719    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"
Sep 10 00:21:41.488891 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.488724    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl" event={"ID":"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10","Type":"ContainerDied","Data":"ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d"}
Sep 10 00:21:41.488891 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.488831    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl" event={"ID":"b0d41660-dbf7-4e2a-8c12-ab3588e8cb10","Type":"ContainerDied","Data":"800f0f41199674424faf76ad4d11c3adff028c12af5c01554f1e242c619fc818"}
Sep 10 00:21:41.488891 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.488852    2556 scope.go:117] "RemoveContainer" containerID="ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d"
Sep 10 00:21:41.498747 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.498726    2556 scope.go:117] "RemoveContainer" containerID="ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d"
Sep 10 00:21:41.499036 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:21:41.499017    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d\": container with ID starting with ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d not found: ID does not exist" containerID="ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d"
Sep 10 00:21:41.499123 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.499044    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d"} err="failed to get container status \"ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d\": rpc error: code = NotFound desc = could not find container \"ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d\": container with ID starting with ca92db1447b03709d4d43b7d94a9e7d6c7cfe3ba15ecef833a885f230c38811d not found: ID does not exist"
Sep 10 00:21:41.509388 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.509369    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"]
Sep 10 00:21:41.512606 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:41.512580    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/sub-test-model-b1baac-kserve-696c449b59-bzppl"]
Sep 10 00:21:42.268333 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:42.268293    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" path="/var/lib/kubelet/pods/b0d41660-dbf7-4e2a-8c12-ab3588e8cb10/volumes"
Sep 10 00:21:42.268881 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:21:42.268850    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b28c8979-95ed-4d14-aa58-0a98a4a77997" path="/var/lib/kubelet/pods/b28c8979-95ed-4d14-aa58-0a98a4a77997/volumes"
Sep 10 00:22:03.037320 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.037285    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"]
Sep 10 00:22:03.037694 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.037664    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28c8979-95ed-4d14-aa58-0a98a4a77997" containerName="main"
Sep 10 00:22:03.037694 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.037675    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28c8979-95ed-4d14-aa58-0a98a4a77997" containerName="main"
Sep 10 00:22:03.037694 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.037687    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8484a80b-5a06-40b5-bd3c-640b6da1dd84" containerName="test-curl-7792-c2e872"
Sep 10 00:22:03.037694 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.037694    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8484a80b-5a06-40b5-bd3c-640b6da1dd84" containerName="test-curl-7792-c2e872"
Sep 10 00:22:03.037838 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.037720    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" containerName="main"
Sep 10 00:22:03.037838 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.037726    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" containerName="main"
Sep 10 00:22:03.037838 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.037780    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8484a80b-5a06-40b5-bd3c-640b6da1dd84" containerName="test-curl-7792-c2e872"
Sep 10 00:22:03.037838 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.037794    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28c8979-95ed-4d14-aa58-0a98a4a77997" containerName="main"
Sep 10 00:22:03.037838 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.037800    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d41660-dbf7-4e2a-8c12-ab3588e8cb10" containerName="main"
Sep 10 00:22:03.040960 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.040938    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.044323 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.044297    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"openshift-service-ca.crt\""
Sep 10 00:22:03.044472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.044339    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"default-dockercfg-jxkgn\""
Sep 10 00:22:03.044472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.044344    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"rate-test-model-b1baac-kserve-self-signed-certs\""
Sep 10 00:22:03.044472 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.044457    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b1baac\"/\"kube-root-ca.crt\""
Sep 10 00:22:03.049311 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.049286    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"]
Sep 10 00:22:03.094921 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.094891    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-dshm\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.094921 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.094924    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-home\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.095101 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.094950    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-model-cache\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.095101 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.095012    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wksv\" (UniqueName: \"kubernetes.io/projected/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-kube-api-access-2wksv\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.095176 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.095108    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tls-certs\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.095176 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.095156    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tmp-dir\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.195711 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.195683    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-dshm\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.195818 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.195714    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-home\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.195818 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.195732    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-model-cache\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.195818 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.195753    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wksv\" (UniqueName: \"kubernetes.io/projected/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-kube-api-access-2wksv\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.195818 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.195788    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tls-certs\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.196006 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.195821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tmp-dir\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.196169 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.196152    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-model-cache\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.196245 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.196218    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tmp-dir\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.196344 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.196323    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-home\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.198042 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.198021    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-dshm\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.198205 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.198189    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tls-certs\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.202676 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.202654    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wksv\" (UniqueName: \"kubernetes.io/projected/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-kube-api-access-2wksv\") pod \"rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") " pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.350555 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.350491    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:03.621245 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:03.621217    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"]
Sep 10 00:22:03.623402 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:22:03.623374    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64e34934_bbbf_4dc4_bb1a_36b471d0ce13.slice/crio-21a990cef709b7016464c1f550fd211a1bddca21d3ecf9e4d9b2bab1cf5aff6c WatchSource:0}: Error finding container 21a990cef709b7016464c1f550fd211a1bddca21d3ecf9e4d9b2bab1cf5aff6c: Status 404 returned error can't find the container with id 21a990cef709b7016464c1f550fd211a1bddca21d3ecf9e4d9b2bab1cf5aff6c
Sep 10 00:22:04.609657 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:04.609614    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j" event={"ID":"64e34934-bbbf-4dc4-bb1a-36b471d0ce13","Type":"ContainerStarted","Data":"a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339"}
Sep 10 00:22:04.609657 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:04.609662    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j" event={"ID":"64e34934-bbbf-4dc4-bb1a-36b471d0ce13","Type":"ContainerStarted","Data":"21a990cef709b7016464c1f550fd211a1bddca21d3ecf9e4d9b2bab1cf5aff6c"}
Sep 10 00:22:04.624556 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:04.624501    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j" podStartSLOduration=1.6244826959999998 podStartE2EDuration="1.624482696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:22:04.623508503 +0000 UTC m=+2004.874644538" watchObservedRunningTime="2026-09-10 00:22:04.624482696 +0000 UTC m=+2004.875618740"
Sep 10 00:22:13.351205 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:13.351169    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:13.351672 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:13.351221    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:13.363769 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:13.363740    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:13.675504 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:13.675420    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:22:45.540016 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:45.539976    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"]
Sep 10 00:22:51.048752 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.048717    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"]
Sep 10 00:22:51.051999 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.051983    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:22:51.056284 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.056267    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-918bbb-dockercfg-7djwz\""
Sep 10 00:22:51.056570 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.056555    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-918bbb\""
Sep 10 00:22:51.065901 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.065876    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"]
Sep 10 00:22:51.081440 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.081420    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"]
Sep 10 00:22:51.085124 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.085108    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:22:51.094925 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.094897    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"]
Sep 10 00:22:51.205518 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.205482    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2cdfae-2571-498c-857b-fe94cd2bccd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x\" (UID: \"4c2cdfae-2571-498c-857b-fe94cd2bccd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:22:51.205686 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.205599    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jpw\" (UniqueName: \"kubernetes.io/projected/4c2cdfae-2571-498c-857b-fe94cd2bccd8-kube-api-access-c9jpw\") pod \"payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x\" (UID: \"4c2cdfae-2571-498c-857b-fe94cd2bccd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:22:51.205686 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.205638    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349746d-5914-4a80-a580-ec20f45b1fd6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn\" (UID: \"6349746d-5914-4a80-a580-ec20f45b1fd6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:22:51.205802 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.205700    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdq5j\" (UniqueName: \"kubernetes.io/projected/6349746d-5914-4a80-a580-ec20f45b1fd6-kube-api-access-gdq5j\") pod \"payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn\" (UID: \"6349746d-5914-4a80-a580-ec20f45b1fd6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:22:51.306562 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.306480    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdq5j\" (UniqueName: \"kubernetes.io/projected/6349746d-5914-4a80-a580-ec20f45b1fd6-kube-api-access-gdq5j\") pod \"payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn\" (UID: \"6349746d-5914-4a80-a580-ec20f45b1fd6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:22:51.306562 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.306557    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2cdfae-2571-498c-857b-fe94cd2bccd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x\" (UID: \"4c2cdfae-2571-498c-857b-fe94cd2bccd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:22:51.306786 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.306605    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jpw\" (UniqueName: \"kubernetes.io/projected/4c2cdfae-2571-498c-857b-fe94cd2bccd8-kube-api-access-c9jpw\") pod \"payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x\" (UID: \"4c2cdfae-2571-498c-857b-fe94cd2bccd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:22:51.306786 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.306630    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349746d-5914-4a80-a580-ec20f45b1fd6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn\" (UID: \"6349746d-5914-4a80-a580-ec20f45b1fd6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:22:51.307383 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.307359    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2cdfae-2571-498c-857b-fe94cd2bccd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x\" (UID: \"4c2cdfae-2571-498c-857b-fe94cd2bccd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:22:51.307383 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.307371    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349746d-5914-4a80-a580-ec20f45b1fd6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn\" (UID: \"6349746d-5914-4a80-a580-ec20f45b1fd6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:22:51.313869 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.313845    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdq5j\" (UniqueName: \"kubernetes.io/projected/6349746d-5914-4a80-a580-ec20f45b1fd6-kube-api-access-gdq5j\") pod \"payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn\" (UID: \"6349746d-5914-4a80-a580-ec20f45b1fd6\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:22:51.313954 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.313845    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jpw\" (UniqueName: \"kubernetes.io/projected/4c2cdfae-2571-498c-857b-fe94cd2bccd8-kube-api-access-c9jpw\") pod \"payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x\" (UID: \"4c2cdfae-2571-498c-857b-fe94cd2bccd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:22:51.361768 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.361744    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:22:51.395789 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.395767    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:22:51.636444 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.636420    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"]
Sep 10 00:22:51.638746 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:22:51.638718    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2cdfae_2571_498c_857b_fe94cd2bccd8.slice/crio-6bbfbad86afb847e3fbd4386d51cf93aa493670b14110bd6798a70c5df971f26 WatchSource:0}: Error finding container 6bbfbad86afb847e3fbd4386d51cf93aa493670b14110bd6798a70c5df971f26: Status 404 returned error can't find the container with id 6bbfbad86afb847e3fbd4386d51cf93aa493670b14110bd6798a70c5df971f26
Sep 10 00:22:51.684334 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.684309    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"]
Sep 10 00:22:51.685832 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:22:51.685810    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6349746d_5914_4a80_a580_ec20f45b1fd6.slice/crio-011de4a9f77599340ad320c617c3d0bfe15a80d8986862117546d039814ebfd9 WatchSource:0}: Error finding container 011de4a9f77599340ad320c617c3d0bfe15a80d8986862117546d039814ebfd9: Status 404 returned error can't find the container with id 011de4a9f77599340ad320c617c3d0bfe15a80d8986862117546d039814ebfd9
Sep 10 00:22:51.864313 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.864277    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn" event={"ID":"6349746d-5914-4a80-a580-ec20f45b1fd6","Type":"ContainerStarted","Data":"a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e"}
Sep 10 00:22:51.864493 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.864335    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn" event={"ID":"6349746d-5914-4a80-a580-ec20f45b1fd6","Type":"ContainerStarted","Data":"011de4a9f77599340ad320c617c3d0bfe15a80d8986862117546d039814ebfd9"}
Sep 10 00:22:51.864493 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.864383    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:22:51.866233 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.866211    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x" event={"ID":"4c2cdfae-2571-498c-857b-fe94cd2bccd8","Type":"ContainerStarted","Data":"31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c"}
Sep 10 00:22:51.866355 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.866248    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x" event={"ID":"4c2cdfae-2571-498c-857b-fe94cd2bccd8","Type":"ContainerStarted","Data":"6bbfbad86afb847e3fbd4386d51cf93aa493670b14110bd6798a70c5df971f26"}
Sep 10 00:22:51.866355 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.866270    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:22:51.882156 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.882087    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn" podStartSLOduration=0.882044835 podStartE2EDuration="882.044835ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:22:51.879034333 +0000 UTC m=+2052.130170378" watchObservedRunningTime="2026-09-10 00:22:51.882044835 +0000 UTC m=+2052.133180881"
Sep 10 00:22:51.893582 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:22:51.893504    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x" podStartSLOduration=0.893486579 podStartE2EDuration="893.486579ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:22:51.892294222 +0000 UTC m=+2052.143430265" watchObservedRunningTime="2026-09-10 00:22:51.893486579 +0000 UTC m=+2052.144622626"
Sep 10 00:23:00.559820 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.559769    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j" podUID="64e34934-bbbf-4dc4-bb1a-36b471d0ce13" containerName="main" containerID="cri-o://a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339" gracePeriod=45
Sep 10 00:23:00.807888 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.807863    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:23:00.892044 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.891959    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wksv\" (UniqueName: \"kubernetes.io/projected/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-kube-api-access-2wksv\") pod \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") "
Sep 10 00:23:00.892044 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.892014    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-home\") pod \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") "
Sep 10 00:23:00.892044 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.892045    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tls-certs\") pod \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") "
Sep 10 00:23:00.892352 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.892083    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tmp-dir\") pod \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") "
Sep 10 00:23:00.892352 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.892143    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-dshm\") pod \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") "
Sep 10 00:23:00.892352 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.892228    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-model-cache\") pod \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\" (UID: \"64e34934-bbbf-4dc4-bb1a-36b471d0ce13\") "
Sep 10 00:23:00.892352 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.892331    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-home" (OuterVolumeSpecName: "home") pod "64e34934-bbbf-4dc4-bb1a-36b471d0ce13" (UID: "64e34934-bbbf-4dc4-bb1a-36b471d0ce13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:23:00.892528 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.892413    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64e34934-bbbf-4dc4-bb1a-36b471d0ce13" (UID: "64e34934-bbbf-4dc4-bb1a-36b471d0ce13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:23:00.892528 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.892508    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-model-cache" (OuterVolumeSpecName: "model-cache") pod "64e34934-bbbf-4dc4-bb1a-36b471d0ce13" (UID: "64e34934-bbbf-4dc4-bb1a-36b471d0ce13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:23:00.892610 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.892587    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:23:00.892610 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.892598    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:23:00.892610 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.892609    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:23:00.894227 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.894205    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-dshm" (OuterVolumeSpecName: "dshm") pod "64e34934-bbbf-4dc4-bb1a-36b471d0ce13" (UID: "64e34934-bbbf-4dc4-bb1a-36b471d0ce13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:23:00.894667 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.894650    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64e34934-bbbf-4dc4-bb1a-36b471d0ce13" (UID: "64e34934-bbbf-4dc4-bb1a-36b471d0ce13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:23:00.894743 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.894661    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-kube-api-access-2wksv" (OuterVolumeSpecName: "kube-api-access-2wksv") pod "64e34934-bbbf-4dc4-bb1a-36b471d0ce13" (UID: "64e34934-bbbf-4dc4-bb1a-36b471d0ce13"). InnerVolumeSpecName "kube-api-access-2wksv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:23:00.917727 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.917697    2556 generic.go:356] "Generic (PLEG): container finished" podID="64e34934-bbbf-4dc4-bb1a-36b471d0ce13" containerID="a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339" exitCode=0
Sep 10 00:23:00.917836 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.917743    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"
Sep 10 00:23:00.917836 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.917782    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j" event={"ID":"64e34934-bbbf-4dc4-bb1a-36b471d0ce13","Type":"ContainerDied","Data":"a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339"}
Sep 10 00:23:00.917836 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.917822    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j" event={"ID":"64e34934-bbbf-4dc4-bb1a-36b471d0ce13","Type":"ContainerDied","Data":"21a990cef709b7016464c1f550fd211a1bddca21d3ecf9e4d9b2bab1cf5aff6c"}
Sep 10 00:23:00.917945 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.917840    2556 scope.go:117] "RemoveContainer" containerID="a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339"
Sep 10 00:23:00.929433 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.929404    2556 scope.go:117] "RemoveContainer" containerID="a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339"
Sep 10 00:23:00.929918 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:23:00.929893    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339\": container with ID starting with a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339 not found: ID does not exist" containerID="a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339"
Sep 10 00:23:00.929989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.929926    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339"} err="failed to get container status \"a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339\": rpc error: code = NotFound desc = could not find container \"a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339\": container with ID starting with a12430b405bb8143460fe204e6bf396ae8da27a64093492621d1d6e0c5803339 not found: ID does not exist"
Sep 10 00:23:00.939190 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.939170    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"]
Sep 10 00:23:00.942270 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.942248    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b1baac/rate-test-model-b1baac-kserve-5fdb8cdf6c-gj72j"]
Sep 10 00:23:00.993797 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.993769    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wksv\" (UniqueName: \"kubernetes.io/projected/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-kube-api-access-2wksv\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:23:00.993797 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.993795    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:23:00.993955 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:00.993805    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64e34934-bbbf-4dc4-bb1a-36b471d0ce13-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:23:02.268716 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:02.268683    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64e34934-bbbf-4dc4-bb1a-36b471d0ce13" path="/var/lib/kubelet/pods/64e34934-bbbf-4dc4-bb1a-36b471d0ce13/volumes"
Sep 10 00:23:02.877826 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:02.877757    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:23:02.877981 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:02.877832    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:23:20.168602 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.168565    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"]
Sep 10 00:23:20.169190 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.169171    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64e34934-bbbf-4dc4-bb1a-36b471d0ce13" containerName="main"
Sep 10 00:23:20.169284 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.169192    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="64e34934-bbbf-4dc4-bb1a-36b471d0ce13" containerName="main"
Sep 10 00:23:20.169342 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.169327    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="64e34934-bbbf-4dc4-bb1a-36b471d0ce13" containerName="main"
Sep 10 00:23:20.172429 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.172409    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:23:20.174579 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.174561    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5c9018\""
Sep 10 00:23:20.174663 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.174631    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5c9018-dockercfg-bsshm\""
Sep 10 00:23:20.184880 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.184860    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"]
Sep 10 00:23:20.209720 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.209695    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"]
Sep 10 00:23:20.213120 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.213104    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:23:20.222861 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.222839    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"]
Sep 10 00:23:20.253760 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.253734    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dbpr\" (UniqueName: \"kubernetes.io/projected/b3130c5c-ba48-4600-b6f6-d40548e491a8-kube-api-access-4dbpr\") pod \"payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9\" (UID: \"b3130c5c-ba48-4600-b6f6-d40548e491a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:23:20.253850 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.253794    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7ab857-48a2-4a7e-aa0d-802783386fe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n\" (UID: \"4e7ab857-48a2-4a7e-aa0d-802783386fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:23:20.253893 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.253845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3130c5c-ba48-4600-b6f6-d40548e491a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9\" (UID: \"b3130c5c-ba48-4600-b6f6-d40548e491a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:23:20.253936 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.253917    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl2lq\" (UniqueName: \"kubernetes.io/projected/4e7ab857-48a2-4a7e-aa0d-802783386fe9-kube-api-access-cl2lq\") pod \"payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n\" (UID: \"4e7ab857-48a2-4a7e-aa0d-802783386fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:23:20.355255 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.355221    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7ab857-48a2-4a7e-aa0d-802783386fe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n\" (UID: \"4e7ab857-48a2-4a7e-aa0d-802783386fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:23:20.355421 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.355260    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3130c5c-ba48-4600-b6f6-d40548e491a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9\" (UID: \"b3130c5c-ba48-4600-b6f6-d40548e491a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:23:20.355421 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.355304    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2lq\" (UniqueName: \"kubernetes.io/projected/4e7ab857-48a2-4a7e-aa0d-802783386fe9-kube-api-access-cl2lq\") pod \"payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n\" (UID: \"4e7ab857-48a2-4a7e-aa0d-802783386fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:23:20.355550 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.355409    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbpr\" (UniqueName: \"kubernetes.io/projected/b3130c5c-ba48-4600-b6f6-d40548e491a8-kube-api-access-4dbpr\") pod \"payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9\" (UID: \"b3130c5c-ba48-4600-b6f6-d40548e491a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:23:20.355964 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.355943    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7ab857-48a2-4a7e-aa0d-802783386fe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n\" (UID: \"4e7ab857-48a2-4a7e-aa0d-802783386fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:23:20.356082 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.356018    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3130c5c-ba48-4600-b6f6-d40548e491a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9\" (UID: \"b3130c5c-ba48-4600-b6f6-d40548e491a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:23:20.362816 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.362791    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2lq\" (UniqueName: \"kubernetes.io/projected/4e7ab857-48a2-4a7e-aa0d-802783386fe9-kube-api-access-cl2lq\") pod \"payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n\" (UID: \"4e7ab857-48a2-4a7e-aa0d-802783386fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:23:20.362816 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.362810    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbpr\" (UniqueName: \"kubernetes.io/projected/b3130c5c-ba48-4600-b6f6-d40548e491a8-kube-api-access-4dbpr\") pod \"payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9\" (UID: \"b3130c5c-ba48-4600-b6f6-d40548e491a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:23:20.483422 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.483380    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:23:20.523226 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.523195    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:23:20.751971 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.751942    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"]
Sep 10 00:23:20.753537 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:23:20.753511    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7ab857_48a2_4a7e_aa0d_802783386fe9.slice/crio-3d6e184b6ea649c7ae3219840ebd0faec401ede2d4fce3233f69e2fa3ea911b1 WatchSource:0}: Error finding container 3d6e184b6ea649c7ae3219840ebd0faec401ede2d4fce3233f69e2fa3ea911b1: Status 404 returned error can't find the container with id 3d6e184b6ea649c7ae3219840ebd0faec401ede2d4fce3233f69e2fa3ea911b1
Sep 10 00:23:20.804172 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:20.804147    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"]
Sep 10 00:23:20.805459 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:23:20.805436    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3130c5c_ba48_4600_b6f6_d40548e491a8.slice/crio-2d036f6b8f9f57d0d036f9a5a3eb1b7ed209bd8052e455b55696de75848c1669 WatchSource:0}: Error finding container 2d036f6b8f9f57d0d036f9a5a3eb1b7ed209bd8052e455b55696de75848c1669: Status 404 returned error can't find the container with id 2d036f6b8f9f57d0d036f9a5a3eb1b7ed209bd8052e455b55696de75848c1669
Sep 10 00:23:21.034507 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:21.034382    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n" event={"ID":"4e7ab857-48a2-4a7e-aa0d-802783386fe9","Type":"ContainerStarted","Data":"bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d"}
Sep 10 00:23:21.034670 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:21.034539    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:23:21.034670 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:21.034573    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n" event={"ID":"4e7ab857-48a2-4a7e-aa0d-802783386fe9","Type":"ContainerStarted","Data":"3d6e184b6ea649c7ae3219840ebd0faec401ede2d4fce3233f69e2fa3ea911b1"}
Sep 10 00:23:21.036117 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:21.036085    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9" event={"ID":"b3130c5c-ba48-4600-b6f6-d40548e491a8","Type":"ContainerStarted","Data":"2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8"}
Sep 10 00:23:21.036234 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:21.036142    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9" event={"ID":"b3130c5c-ba48-4600-b6f6-d40548e491a8","Type":"ContainerStarted","Data":"2d036f6b8f9f57d0d036f9a5a3eb1b7ed209bd8052e455b55696de75848c1669"}
Sep 10 00:23:21.036234 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:21.036164    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:23:21.050461 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:21.050419    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n" podStartSLOduration=1.05040554 podStartE2EDuration="1.05040554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:23:21.048556359 +0000 UTC m=+2081.299692405" watchObservedRunningTime="2026-09-10 00:23:21.05040554 +0000 UTC m=+2081.301541583"
Sep 10 00:23:21.064007 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:21.063958    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9" podStartSLOduration=1.063940919 podStartE2EDuration="1.063940919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:23:21.062122809 +0000 UTC m=+2081.313258881" watchObservedRunningTime="2026-09-10 00:23:21.063940919 +0000 UTC m=+2081.315076967"
Sep 10 00:23:32.043172 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:32.043142    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:23:32.043705 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:32.043213    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:23:40.697427 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:40.697390    2556 scope.go:117] "RemoveContainer" containerID="0b7cbfddd7946faac6149ed2a3c98d317c1e53ab3a12d2e34287d40977a9a94c"
Sep 10 00:23:40.707484 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:40.707460    2556 scope.go:117] "RemoveContainer" containerID="bc34750a7f76d2d9cc202e89e7c14f5ab211f36b9aaddd44d0848a60e58ad3b2"
Sep 10 00:23:48.834367 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:23:48.834329    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9_b3130c5c-ba48-4600-b6f6-d40548e491a8/payload-processing/0.log"
Sep 10 00:24:02.272778 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.272746    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"]
Sep 10 00:24:02.276813 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.276791    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.278830 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.278804    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"]
Sep 10 00:24:02.280285 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.280126    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-918bbb\"/\"kube-root-ca.crt\""
Sep 10 00:24:02.280285 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.280147    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-918bbb\"/\"default-dockercfg-mk89g\""
Sep 10 00:24:02.280464 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.280380    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-918bbb\"/\"openshift-service-ca.crt\""
Sep 10 00:24:02.280597 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.280574    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-918bbb\"/\"ipp-route-918bbb-kserve-self-signed-certs\""
Sep 10 00:24:02.305744 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.305723    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14837297-91a9-440a-865f-d949d9c1d740-tls-certs\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.305873 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.305758    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-dshm\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.305873 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.305784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-tmp-dir\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.305982 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.305864    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtg6\" (UniqueName: \"kubernetes.io/projected/14837297-91a9-440a-865f-d949d9c1d740-kube-api-access-pbtg6\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.305982 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.305914    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-model-cache\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.305982 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.305975    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-home\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.407330 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.407257    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-tmp-dir\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.407330 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.407297    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtg6\" (UniqueName: \"kubernetes.io/projected/14837297-91a9-440a-865f-d949d9c1d740-kube-api-access-pbtg6\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.407537 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.407427    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-model-cache\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.407537 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.407503    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-home\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.407650 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.407634    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14837297-91a9-440a-865f-d949d9c1d740-tls-certs\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.407712 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.407682    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-tmp-dir\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.407765 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.407688    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-dshm\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.407765 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.407718    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-model-cache\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.407877 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.407820    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-home\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.409984 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.409959    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-dshm\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.410142 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.410124    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14837297-91a9-440a-865f-d949d9c1d740-tls-certs\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.414632 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.414609    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtg6\" (UniqueName: \"kubernetes.io/projected/14837297-91a9-440a-865f-d949d9c1d740-kube-api-access-pbtg6\") pod \"ipp-route-918bbb-kserve-7f6d9f84c-927g9\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") " pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.587144 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.587116    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:02.860804 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:02.860775    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"]
Sep 10 00:24:02.862462 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:24:02.862433    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14837297_91a9_440a_865f_d949d9c1d740.slice/crio-e94c92fc8695dbd01e915744ab7f6a1cc71fd738356df469b48764830415210a WatchSource:0}: Error finding container e94c92fc8695dbd01e915744ab7f6a1cc71fd738356df469b48764830415210a: Status 404 returned error can't find the container with id e94c92fc8695dbd01e915744ab7f6a1cc71fd738356df469b48764830415210a
Sep 10 00:24:03.273237 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:03.273204    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9" event={"ID":"14837297-91a9-440a-865f-d949d9c1d740","Type":"ContainerStarted","Data":"7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273"}
Sep 10 00:24:03.273237 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:03.273242    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9" event={"ID":"14837297-91a9-440a-865f-d949d9c1d740","Type":"ContainerStarted","Data":"e94c92fc8695dbd01e915744ab7f6a1cc71fd738356df469b48764830415210a"}
Sep 10 00:24:03.289607 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:03.289558    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9" podStartSLOduration=1.289544745 podStartE2EDuration="1.289544745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:24:03.287120851 +0000 UTC m=+2123.538256893" watchObservedRunningTime="2026-09-10 00:24:03.289544745 +0000 UTC m=+2123.540680787"
Sep 10 00:24:12.587603 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:12.587562    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:12.588023 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:12.587617    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:12.601146 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:12.601117    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:13.334986 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:13.334955    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:24:24.600529 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:24.600496    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn_6349746d-5914-4a80-a580-ec20f45b1fd6/payload-processing/0.log"
Sep 10 00:24:24.832112 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:24.832086    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9_b3130c5c-ba48-4600-b6f6-d40548e491a8/payload-processing/0.log"
Sep 10 00:24:30.688032 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:30.687996    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"]
Sep 10 00:24:30.691794 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:30.691778    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:24:30.694378 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:30.694303    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-70937b-dockercfg-wmrr4\""
Sep 10 00:24:30.694378 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:30.694344    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-70937b\""
Sep 10 00:24:30.700398 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:30.700120    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"]
Sep 10 00:24:30.853925 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:30.853880    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcwrx\" (UniqueName: \"kubernetes.io/projected/fb930844-2494-4488-aab3-38033d580935-kube-api-access-xcwrx\") pod \"payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z\" (UID: \"fb930844-2494-4488-aab3-38033d580935\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:24:30.854134 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:30.854040    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb930844-2494-4488-aab3-38033d580935-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z\" (UID: \"fb930844-2494-4488-aab3-38033d580935\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:24:30.954713 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:30.954678    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb930844-2494-4488-aab3-38033d580935-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z\" (UID: \"fb930844-2494-4488-aab3-38033d580935\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:24:30.954905 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:30.954721    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcwrx\" (UniqueName: \"kubernetes.io/projected/fb930844-2494-4488-aab3-38033d580935-kube-api-access-xcwrx\") pod \"payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z\" (UID: \"fb930844-2494-4488-aab3-38033d580935\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:24:30.955436 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:30.955414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb930844-2494-4488-aab3-38033d580935-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z\" (UID: \"fb930844-2494-4488-aab3-38033d580935\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:24:30.963267 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:30.963241    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcwrx\" (UniqueName: \"kubernetes.io/projected/fb930844-2494-4488-aab3-38033d580935-kube-api-access-xcwrx\") pod \"payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z\" (UID: \"fb930844-2494-4488-aab3-38033d580935\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:24:31.004427 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:31.004402    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:24:31.275915 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:31.275883    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"]
Sep 10 00:24:31.277234 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:24:31.277202    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb930844_2494_4488_aab3_38033d580935.slice/crio-35a6471dbe313a75af7ccc9022a97f5d9b8d48deea9dc049caa2a1286cf5c0b9 WatchSource:0}: Error finding container 35a6471dbe313a75af7ccc9022a97f5d9b8d48deea9dc049caa2a1286cf5c0b9: Status 404 returned error can't find the container with id 35a6471dbe313a75af7ccc9022a97f5d9b8d48deea9dc049caa2a1286cf5c0b9
Sep 10 00:24:31.423272 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:31.423228    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z" event={"ID":"fb930844-2494-4488-aab3-38033d580935","Type":"ContainerStarted","Data":"37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02"}
Sep 10 00:24:31.423459 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:31.423288    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z" event={"ID":"fb930844-2494-4488-aab3-38033d580935","Type":"ContainerStarted","Data":"35a6471dbe313a75af7ccc9022a97f5d9b8d48deea9dc049caa2a1286cf5c0b9"}
Sep 10 00:24:31.423459 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:31.423361    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:24:31.439545 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:31.439501    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z" podStartSLOduration=1.439486721 podStartE2EDuration="1.439486721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:24:31.436541674 +0000 UTC m=+2151.687677719" watchObservedRunningTime="2026-09-10 00:24:31.439486721 +0000 UTC m=+2151.690622836"
Sep 10 00:24:40.738904 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:40.738877    2556 scope.go:117] "RemoveContainer" containerID="ec3df29ed66a9cda033c7e2e4efc0217f981bd7b8286cda60903eb4a3d934314"
Sep 10 00:24:40.749169 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:40.749149    2556 scope.go:117] "RemoveContainer" containerID="36a53d22703886975aac629b50e5c8b0c20d990f3fcd10de140f7adc55ad8b6a"
Sep 10 00:24:40.757066 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:40.757038    2556 scope.go:117] "RemoveContainer" containerID="fd71361ac7905024110105ac961ef3ce1ee4355dc0d0d39a7daa6c71e76bba83"
Sep 10 00:24:42.430561 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:42.430530    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:24:54.403901 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:54.403866    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"]
Sep 10 00:24:54.407377 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:54.407361    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"
Sep 10 00:24:54.409636 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:54.409611    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:24:54.409760 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:54.409667    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:24:54.413933 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:54.413909    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"]
Sep 10 00:24:54.440624 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:54.440597    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4762b84b-984b-457d-8fa5-a5500022dd5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg\" (UID: \"4762b84b-984b-457d-8fa5-a5500022dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"
Sep 10 00:24:54.542079 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:54.542040    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4762b84b-984b-457d-8fa5-a5500022dd5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg\" (UID: \"4762b84b-984b-457d-8fa5-a5500022dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"
Sep 10 00:24:54.542665 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:54.542643    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4762b84b-984b-457d-8fa5-a5500022dd5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg\" (UID: \"4762b84b-984b-457d-8fa5-a5500022dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"
Sep 10 00:24:54.719533 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:54.719512    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"
Sep 10 00:24:54.988220 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:54.988194    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"]
Sep 10 00:24:54.990368 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:24:54.990342    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4762b84b_984b_457d_8fa5_a5500022dd5f.slice/crio-c1024d08f4a3810dacfc36d2b631f13d05e9e4089accb669615ae75b2acd69d4 WatchSource:0}: Error finding container c1024d08f4a3810dacfc36d2b631f13d05e9e4089accb669615ae75b2acd69d4: Status 404 returned error can't find the container with id c1024d08f4a3810dacfc36d2b631f13d05e9e4089accb669615ae75b2acd69d4
Sep 10 00:24:55.550182 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:55.550097    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg" event={"ID":"4762b84b-984b-457d-8fa5-a5500022dd5f","Type":"ContainerStarted","Data":"b620445eb8d3893db770e143f6d2928eac1e67e6ee06dd17a9db6def900a337b"}
Sep 10 00:24:55.550182 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:55.550153    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg" event={"ID":"4762b84b-984b-457d-8fa5-a5500022dd5f","Type":"ContainerStarted","Data":"c1024d08f4a3810dacfc36d2b631f13d05e9e4089accb669615ae75b2acd69d4"}
Sep 10 00:24:55.564032 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:55.563877    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg" podStartSLOduration=1.563859485 podStartE2EDuration="1.563859485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:24:55.563708675 +0000 UTC m=+2175.814844719" watchObservedRunningTime="2026-09-10 00:24:55.563859485 +0000 UTC m=+2175.814995529"
Sep 10 00:24:56.556615 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:56.556578    2556 generic.go:356] "Generic (PLEG): container finished" podID="4762b84b-984b-457d-8fa5-a5500022dd5f" containerID="b620445eb8d3893db770e143f6d2928eac1e67e6ee06dd17a9db6def900a337b" exitCode=0
Sep 10 00:24:56.556987 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:56.556666    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg" event={"ID":"4762b84b-984b-457d-8fa5-a5500022dd5f","Type":"ContainerDied","Data":"b620445eb8d3893db770e143f6d2928eac1e67e6ee06dd17a9db6def900a337b"}
Sep 10 00:24:57.696525 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:57.696502    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"
Sep 10 00:24:57.769280 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:57.769247    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4762b84b-984b-457d-8fa5-a5500022dd5f-maas-api-service-ca\") pod \"4762b84b-984b-457d-8fa5-a5500022dd5f\" (UID: \"4762b84b-984b-457d-8fa5-a5500022dd5f\") "
Sep 10 00:24:57.769582 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:57.769559    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4762b84b-984b-457d-8fa5-a5500022dd5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4762b84b-984b-457d-8fa5-a5500022dd5f" (UID: "4762b84b-984b-457d-8fa5-a5500022dd5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:24:57.870178 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:57.870110    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4762b84b-984b-457d-8fa5-a5500022dd5f-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:24:58.568451 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:58.568375    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"
Sep 10 00:24:58.568451 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:58.568418    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg" event={"ID":"4762b84b-984b-457d-8fa5-a5500022dd5f","Type":"ContainerDied","Data":"c1024d08f4a3810dacfc36d2b631f13d05e9e4089accb669615ae75b2acd69d4"}
Sep 10 00:24:58.568624 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:24:58.568465    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1024d08f4a3810dacfc36d2b631f13d05e9e4089accb669615ae75b2acd69d4"
Sep 10 00:25:00.533543 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:00.533507    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"]
Sep 10 00:25:00.534082 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:00.533849    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z" podUID="fb930844-2494-4488-aab3-38033d580935" containerName="payload-processing" containerID="cri-o://37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02" gracePeriod=30
Sep 10 00:25:00.681275 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:00.681240    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"]
Sep 10 00:25:00.684604 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:00.684576    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-70937b-wxrrg"]
Sep 10 00:25:00.789658 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:00.789608    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:25:00.896474 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:00.896447    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcwrx\" (UniqueName: \"kubernetes.io/projected/fb930844-2494-4488-aab3-38033d580935-kube-api-access-xcwrx\") pod \"fb930844-2494-4488-aab3-38033d580935\" (UID: \"fb930844-2494-4488-aab3-38033d580935\") "
Sep 10 00:25:00.896651 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:00.896560    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb930844-2494-4488-aab3-38033d580935-plugins-config-volume\") pod \"fb930844-2494-4488-aab3-38033d580935\" (UID: \"fb930844-2494-4488-aab3-38033d580935\") "
Sep 10 00:25:00.896941 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:00.896911    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb930844-2494-4488-aab3-38033d580935-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb930844-2494-4488-aab3-38033d580935" (UID: "fb930844-2494-4488-aab3-38033d580935"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:25:00.898614 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:00.898593    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb930844-2494-4488-aab3-38033d580935-kube-api-access-xcwrx" (OuterVolumeSpecName: "kube-api-access-xcwrx") pod "fb930844-2494-4488-aab3-38033d580935" (UID: "fb930844-2494-4488-aab3-38033d580935"). InnerVolumeSpecName "kube-api-access-xcwrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:25:00.997998 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:00.997971    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcwrx\" (UniqueName: \"kubernetes.io/projected/fb930844-2494-4488-aab3-38033d580935-kube-api-access-xcwrx\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:00.997998 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:00.997995    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb930844-2494-4488-aab3-38033d580935-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:01.586367 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:01.586332    2556 generic.go:356] "Generic (PLEG): container finished" podID="fb930844-2494-4488-aab3-38033d580935" containerID="37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02" exitCode=0
Sep 10 00:25:01.586844 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:01.586387    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"
Sep 10 00:25:01.586844 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:01.586422    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z" event={"ID":"fb930844-2494-4488-aab3-38033d580935","Type":"ContainerDied","Data":"37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02"}
Sep 10 00:25:01.586844 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:01.586485    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z" event={"ID":"fb930844-2494-4488-aab3-38033d580935","Type":"ContainerDied","Data":"35a6471dbe313a75af7ccc9022a97f5d9b8d48deea9dc049caa2a1286cf5c0b9"}
Sep 10 00:25:01.586844 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:01.586521    2556 scope.go:117] "RemoveContainer" containerID="37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02"
Sep 10 00:25:01.595761 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:01.595741    2556 scope.go:117] "RemoveContainer" containerID="37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02"
Sep 10 00:25:01.596005 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:25:01.595990    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02\": container with ID starting with 37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02 not found: ID does not exist" containerID="37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02"
Sep 10 00:25:01.596042 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:01.596012    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02"} err="failed to get container status \"37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02\": rpc error: code = NotFound desc = could not find container \"37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02\": container with ID starting with 37f7a08d5f71457668deabf4b1ff94f9e92126ab9e558305978c0a36b9412f02 not found: ID does not exist"
Sep 10 00:25:01.607309 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:01.607286    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"]
Sep 10 00:25:01.610009 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:01.609989    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-70937b-77b79fd9ff-5fs7z"]
Sep 10 00:25:02.268909 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:02.268871    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4762b84b-984b-457d-8fa5-a5500022dd5f" path="/var/lib/kubelet/pods/4762b84b-984b-457d-8fa5-a5500022dd5f/volumes"
Sep 10 00:25:02.269316 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:02.269301    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb930844-2494-4488-aab3-38033d580935" path="/var/lib/kubelet/pods/fb930844-2494-4488-aab3-38033d580935/volumes"
Sep 10 00:25:10.368960 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.368912    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"]
Sep 10 00:25:10.369612 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.369592    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4762b84b-984b-457d-8fa5-a5500022dd5f" containerName="revoke-keys"
Sep 10 00:25:10.369663 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.369617    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4762b84b-984b-457d-8fa5-a5500022dd5f" containerName="revoke-keys"
Sep 10 00:25:10.369700 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.369668    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb930844-2494-4488-aab3-38033d580935" containerName="payload-processing"
Sep 10 00:25:10.369700 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.369678    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb930844-2494-4488-aab3-38033d580935" containerName="payload-processing"
Sep 10 00:25:10.369781 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.369769    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4762b84b-984b-457d-8fa5-a5500022dd5f" containerName="revoke-keys"
Sep 10 00:25:10.369815 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.369790    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb930844-2494-4488-aab3-38033d580935" containerName="payload-processing"
Sep 10 00:25:10.373766 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.373740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"
Sep 10 00:25:10.376366 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.376340    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:25:10.376544 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.376373    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:25:10.376989 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.376952    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"]
Sep 10 00:25:10.484073 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.484026    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/604c0c29-4a54-4ed2-9087-a8f627002be2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw\" (UID: \"604c0c29-4a54-4ed2-9087-a8f627002be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"
Sep 10 00:25:10.584786 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.584751    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/604c0c29-4a54-4ed2-9087-a8f627002be2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw\" (UID: \"604c0c29-4a54-4ed2-9087-a8f627002be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"
Sep 10 00:25:10.585356 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.585338    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/604c0c29-4a54-4ed2-9087-a8f627002be2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw\" (UID: \"604c0c29-4a54-4ed2-9087-a8f627002be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"
Sep 10 00:25:10.686285 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.686212    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"
Sep 10 00:25:10.953641 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:10.953583    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"]
Sep 10 00:25:10.955938 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:25:10.955913    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604c0c29_4a54_4ed2_9087_a8f627002be2.slice/crio-d02d373b0e2af549c08a36d1044ec9130e69378cdd7e55a9ec2a685c9bcc97f7 WatchSource:0}: Error finding container d02d373b0e2af549c08a36d1044ec9130e69378cdd7e55a9ec2a685c9bcc97f7: Status 404 returned error can't find the container with id d02d373b0e2af549c08a36d1044ec9130e69378cdd7e55a9ec2a685c9bcc97f7
Sep 10 00:25:11.640132 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:11.640037    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw" event={"ID":"604c0c29-4a54-4ed2-9087-a8f627002be2","Type":"ContainerStarted","Data":"3fcdfb91b3f68a3b0dc50e2e75355c3de40ffe5c9740440bc285057b5de3473a"}
Sep 10 00:25:11.640132 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:11.640105    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw" event={"ID":"604c0c29-4a54-4ed2-9087-a8f627002be2","Type":"ContainerStarted","Data":"d02d373b0e2af549c08a36d1044ec9130e69378cdd7e55a9ec2a685c9bcc97f7"}
Sep 10 00:25:11.654639 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:11.654590    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw" podStartSLOduration=1.654575254 podStartE2EDuration="1.654575254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:25:11.653960976 +0000 UTC m=+2191.905097019" watchObservedRunningTime="2026-09-10 00:25:11.654575254 +0000 UTC m=+2191.905711297"
Sep 10 00:25:12.646854 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:12.646822    2556 generic.go:356] "Generic (PLEG): container finished" podID="604c0c29-4a54-4ed2-9087-a8f627002be2" containerID="3fcdfb91b3f68a3b0dc50e2e75355c3de40ffe5c9740440bc285057b5de3473a" exitCode=0
Sep 10 00:25:12.647346 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:12.646906    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw" event={"ID":"604c0c29-4a54-4ed2-9087-a8f627002be2","Type":"ContainerDied","Data":"3fcdfb91b3f68a3b0dc50e2e75355c3de40ffe5c9740440bc285057b5de3473a"}
Sep 10 00:25:13.788692 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:13.788669    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"
Sep 10 00:25:13.910801 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:13.910730    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/604c0c29-4a54-4ed2-9087-a8f627002be2-maas-api-service-ca\") pod \"604c0c29-4a54-4ed2-9087-a8f627002be2\" (UID: \"604c0c29-4a54-4ed2-9087-a8f627002be2\") "
Sep 10 00:25:13.911068 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:13.911029    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604c0c29-4a54-4ed2-9087-a8f627002be2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "604c0c29-4a54-4ed2-9087-a8f627002be2" (UID: "604c0c29-4a54-4ed2-9087-a8f627002be2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:25:14.012215 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:14.012190    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/604c0c29-4a54-4ed2-9087-a8f627002be2-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:14.662956 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:14.662870    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"
Sep 10 00:25:14.662956 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:14.662904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw" event={"ID":"604c0c29-4a54-4ed2-9087-a8f627002be2","Type":"ContainerDied","Data":"d02d373b0e2af549c08a36d1044ec9130e69378cdd7e55a9ec2a685c9bcc97f7"}
Sep 10 00:25:14.663180 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:14.662965    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d02d373b0e2af549c08a36d1044ec9130e69378cdd7e55a9ec2a685c9bcc97f7"
Sep 10 00:25:25.906512 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:25.906479    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"]
Sep 10 00:25:25.906971 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:25.906731    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn" podUID="6349746d-5914-4a80-a580-ec20f45b1fd6" containerName="payload-processing" containerID="cri-o://a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e" gracePeriod=30
Sep 10 00:25:25.919670 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:25.919640    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"]
Sep 10 00:25:25.920428 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:25.920396    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x" podUID="4c2cdfae-2571-498c-857b-fe94cd2bccd8" containerName="payload-pre-processing" containerID="cri-o://31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c" gracePeriod=30
Sep 10 00:25:26.060789 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.060725    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"]
Sep 10 00:25:26.063291 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.063261    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-918bbb-fjvqw"]
Sep 10 00:25:26.182716 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.182690    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:25:26.214062 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.214030    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:25:26.269691 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.269664    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="604c0c29-4a54-4ed2-9087-a8f627002be2" path="/var/lib/kubelet/pods/604c0c29-4a54-4ed2-9087-a8f627002be2/volumes"
Sep 10 00:25:26.306971 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.306949    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2cdfae-2571-498c-857b-fe94cd2bccd8-plugins-config-volume\") pod \"4c2cdfae-2571-498c-857b-fe94cd2bccd8\" (UID: \"4c2cdfae-2571-498c-857b-fe94cd2bccd8\") "
Sep 10 00:25:26.307111 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.307013    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9jpw\" (UniqueName: \"kubernetes.io/projected/4c2cdfae-2571-498c-857b-fe94cd2bccd8-kube-api-access-c9jpw\") pod \"4c2cdfae-2571-498c-857b-fe94cd2bccd8\" (UID: \"4c2cdfae-2571-498c-857b-fe94cd2bccd8\") "
Sep 10 00:25:26.307182 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.307125    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdq5j\" (UniqueName: \"kubernetes.io/projected/6349746d-5914-4a80-a580-ec20f45b1fd6-kube-api-access-gdq5j\") pod \"6349746d-5914-4a80-a580-ec20f45b1fd6\" (UID: \"6349746d-5914-4a80-a580-ec20f45b1fd6\") "
Sep 10 00:25:26.307182 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.307154    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349746d-5914-4a80-a580-ec20f45b1fd6-plugins-config-volume\") pod \"6349746d-5914-4a80-a580-ec20f45b1fd6\" (UID: \"6349746d-5914-4a80-a580-ec20f45b1fd6\") "
Sep 10 00:25:26.307405 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.307380    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2cdfae-2571-498c-857b-fe94cd2bccd8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c2cdfae-2571-498c-857b-fe94cd2bccd8" (UID: "4c2cdfae-2571-498c-857b-fe94cd2bccd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:25:26.307587 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.307564    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6349746d-5914-4a80-a580-ec20f45b1fd6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6349746d-5914-4a80-a580-ec20f45b1fd6" (UID: "6349746d-5914-4a80-a580-ec20f45b1fd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:25:26.309260 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.309238    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2cdfae-2571-498c-857b-fe94cd2bccd8-kube-api-access-c9jpw" (OuterVolumeSpecName: "kube-api-access-c9jpw") pod "4c2cdfae-2571-498c-857b-fe94cd2bccd8" (UID: "4c2cdfae-2571-498c-857b-fe94cd2bccd8"). InnerVolumeSpecName "kube-api-access-c9jpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:25:26.309343 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.309262    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6349746d-5914-4a80-a580-ec20f45b1fd6-kube-api-access-gdq5j" (OuterVolumeSpecName: "kube-api-access-gdq5j") pod "6349746d-5914-4a80-a580-ec20f45b1fd6" (UID: "6349746d-5914-4a80-a580-ec20f45b1fd6"). InnerVolumeSpecName "kube-api-access-gdq5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:25:26.408323 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.408300    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jpw\" (UniqueName: \"kubernetes.io/projected/4c2cdfae-2571-498c-857b-fe94cd2bccd8-kube-api-access-c9jpw\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:26.408323 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.408320    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdq5j\" (UniqueName: \"kubernetes.io/projected/6349746d-5914-4a80-a580-ec20f45b1fd6-kube-api-access-gdq5j\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:26.408447 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.408330    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349746d-5914-4a80-a580-ec20f45b1fd6-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:26.408447 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.408339    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2cdfae-2571-498c-857b-fe94cd2bccd8-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:26.732090 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.732039    2556 generic.go:356] "Generic (PLEG): container finished" podID="6349746d-5914-4a80-a580-ec20f45b1fd6" containerID="a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e" exitCode=0
Sep 10 00:25:26.732261 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.732102    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"
Sep 10 00:25:26.732261 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.732128    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn" event={"ID":"6349746d-5914-4a80-a580-ec20f45b1fd6","Type":"ContainerDied","Data":"a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e"}
Sep 10 00:25:26.732261 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.732189    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn" event={"ID":"6349746d-5914-4a80-a580-ec20f45b1fd6","Type":"ContainerDied","Data":"011de4a9f77599340ad320c617c3d0bfe15a80d8986862117546d039814ebfd9"}
Sep 10 00:25:26.732261 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.732225    2556 scope.go:117] "RemoveContainer" containerID="a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e"
Sep 10 00:25:26.733545 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.733522    2556 generic.go:356] "Generic (PLEG): container finished" podID="4c2cdfae-2571-498c-857b-fe94cd2bccd8" containerID="31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c" exitCode=0
Sep 10 00:25:26.733659 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.733562    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"
Sep 10 00:25:26.733659 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.733573    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x" event={"ID":"4c2cdfae-2571-498c-857b-fe94cd2bccd8","Type":"ContainerDied","Data":"31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c"}
Sep 10 00:25:26.733659 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.733628    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x" event={"ID":"4c2cdfae-2571-498c-857b-fe94cd2bccd8","Type":"ContainerDied","Data":"6bbfbad86afb847e3fbd4386d51cf93aa493670b14110bd6798a70c5df971f26"}
Sep 10 00:25:26.741612 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.741596    2556 scope.go:117] "RemoveContainer" containerID="a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e"
Sep 10 00:25:26.741894 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:25:26.741871    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e\": container with ID starting with a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e not found: ID does not exist" containerID="a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e"
Sep 10 00:25:26.741993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.741902    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e"} err="failed to get container status \"a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e\": rpc error: code = NotFound desc = could not find container \"a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e\": container with ID starting with a74955a153c192e459e5a9ed3c6545a27f907c49044991dd1aeee58ef25a024e not found: ID does not exist"
Sep 10 00:25:26.741993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.741923    2556 scope.go:117] "RemoveContainer" containerID="31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c"
Sep 10 00:25:26.750100 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.750085    2556 scope.go:117] "RemoveContainer" containerID="31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c"
Sep 10 00:25:26.750333 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:25:26.750316    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c\": container with ID starting with 31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c not found: ID does not exist" containerID="31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c"
Sep 10 00:25:26.750396 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.750338    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c"} err="failed to get container status \"31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c\": rpc error: code = NotFound desc = could not find container \"31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c\": container with ID starting with 31cffcc868d3c98207bba22311b4b43894f6409bf010ea408a9be5ef868f847c not found: ID does not exist"
Sep 10 00:25:26.755867 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.755847    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"]
Sep 10 00:25:26.761493 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.761469    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-918bbb-685f64b8fc-5xtbn"]
Sep 10 00:25:26.769956 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.769937    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"]
Sep 10 00:25:26.772859 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:26.772837    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-918bbb-567cdb9b8b-7564x"]
Sep 10 00:25:28.268671 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:28.268638    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c2cdfae-2571-498c-857b-fe94cd2bccd8" path="/var/lib/kubelet/pods/4c2cdfae-2571-498c-857b-fe94cd2bccd8/volumes"
Sep 10 00:25:28.269166 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:28.269001    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6349746d-5914-4a80-a580-ec20f45b1fd6" path="/var/lib/kubelet/pods/6349746d-5914-4a80-a580-ec20f45b1fd6/volumes"
Sep 10 00:25:31.421213 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:31.421182    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"]
Sep 10 00:25:40.783522 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:40.783486    2556 scope.go:117] "RemoveContainer" containerID="11b8633a16c98639fd2f9482cfd72b1097e3aa3453c9c8cbbfbad2f88e95ec48"
Sep 10 00:25:46.440819 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.440780    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9" podUID="14837297-91a9-440a-865f-d949d9c1d740" containerName="main" containerID="cri-o://7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273" gracePeriod=45
Sep 10 00:25:46.677625 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.677600    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:25:46.780523 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.780482    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-dshm\") pod \"14837297-91a9-440a-865f-d949d9c1d740\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") "
Sep 10 00:25:46.780689 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.780586    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-tmp-dir\") pod \"14837297-91a9-440a-865f-d949d9c1d740\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") "
Sep 10 00:25:46.780689 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.780622    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14837297-91a9-440a-865f-d949d9c1d740-tls-certs\") pod \"14837297-91a9-440a-865f-d949d9c1d740\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") "
Sep 10 00:25:46.780689 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.780678    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbtg6\" (UniqueName: \"kubernetes.io/projected/14837297-91a9-440a-865f-d949d9c1d740-kube-api-access-pbtg6\") pod \"14837297-91a9-440a-865f-d949d9c1d740\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") "
Sep 10 00:25:46.780858 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.780720    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-model-cache\") pod \"14837297-91a9-440a-865f-d949d9c1d740\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") "
Sep 10 00:25:46.780858 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.780775    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-home\") pod \"14837297-91a9-440a-865f-d949d9c1d740\" (UID: \"14837297-91a9-440a-865f-d949d9c1d740\") "
Sep 10 00:25:46.780979 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.780851    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14837297-91a9-440a-865f-d949d9c1d740" (UID: "14837297-91a9-440a-865f-d949d9c1d740"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:25:46.781040 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.781001    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-model-cache" (OuterVolumeSpecName: "model-cache") pod "14837297-91a9-440a-865f-d949d9c1d740" (UID: "14837297-91a9-440a-865f-d949d9c1d740"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:25:46.781140 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.781096    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-home" (OuterVolumeSpecName: "home") pod "14837297-91a9-440a-865f-d949d9c1d740" (UID: "14837297-91a9-440a-865f-d949d9c1d740"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:25:46.781200 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.781167    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-tmp-dir\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:46.781200 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.781182    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-model-cache\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:46.781200 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.781191    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-home\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:46.782807 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.782783    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14837297-91a9-440a-865f-d949d9c1d740-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14837297-91a9-440a-865f-d949d9c1d740" (UID: "14837297-91a9-440a-865f-d949d9c1d740"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:25:46.782991 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.782973    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-dshm" (OuterVolumeSpecName: "dshm") pod "14837297-91a9-440a-865f-d949d9c1d740" (UID: "14837297-91a9-440a-865f-d949d9c1d740"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 00:25:46.783047 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.782976    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14837297-91a9-440a-865f-d949d9c1d740-kube-api-access-pbtg6" (OuterVolumeSpecName: "kube-api-access-pbtg6") pod "14837297-91a9-440a-865f-d949d9c1d740" (UID: "14837297-91a9-440a-865f-d949d9c1d740"). InnerVolumeSpecName "kube-api-access-pbtg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:25:46.841263 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.841225    2556 generic.go:356] "Generic (PLEG): container finished" podID="14837297-91a9-440a-865f-d949d9c1d740" containerID="7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273" exitCode=0
Sep 10 00:25:46.841356 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.841269    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"
Sep 10 00:25:46.841356 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.841314    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9" event={"ID":"14837297-91a9-440a-865f-d949d9c1d740","Type":"ContainerDied","Data":"7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273"}
Sep 10 00:25:46.841429 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.841361    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9" event={"ID":"14837297-91a9-440a-865f-d949d9c1d740","Type":"ContainerDied","Data":"e94c92fc8695dbd01e915744ab7f6a1cc71fd738356df469b48764830415210a"}
Sep 10 00:25:46.841429 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.841384    2556 scope.go:117] "RemoveContainer" containerID="7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273"
Sep 10 00:25:46.850737 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.850720    2556 scope.go:117] "RemoveContainer" containerID="7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273"
Sep 10 00:25:46.851003 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:25:46.850985    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273\": container with ID starting with 7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273 not found: ID does not exist" containerID="7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273"
Sep 10 00:25:46.851091 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.851011    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273"} err="failed to get container status \"7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273\": rpc error: code = NotFound desc = could not find container \"7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273\": container with ID starting with 7c4584ec1dcacf2f8bc12857a927e00d8f01b38a0bd072d9a38c61dbab126273 not found: ID does not exist"
Sep 10 00:25:46.862206 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.862164    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"]
Sep 10 00:25:46.864346 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.864325    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-918bbb/ipp-route-918bbb-kserve-7f6d9f84c-927g9"]
Sep 10 00:25:46.882320 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.882298    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14837297-91a9-440a-865f-d949d9c1d740-tls-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:46.882320 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.882320    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbtg6\" (UniqueName: \"kubernetes.io/projected/14837297-91a9-440a-865f-d949d9c1d740-kube-api-access-pbtg6\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:46.882443 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:46.882330    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14837297-91a9-440a-865f-d949d9c1d740-dshm\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:25:48.268251 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:25:48.268207    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14837297-91a9-440a-865f-d949d9c1d740" path="/var/lib/kubelet/pods/14837297-91a9-440a-865f-d949d9c1d740/volumes"
Sep 10 00:26:20.196688 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.196612    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"]
Sep 10 00:26:20.197225 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197151    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14837297-91a9-440a-865f-d949d9c1d740" containerName="main"
Sep 10 00:26:20.197225 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197169    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="14837297-91a9-440a-865f-d949d9c1d740" containerName="main"
Sep 10 00:26:20.197225 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197199    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6349746d-5914-4a80-a580-ec20f45b1fd6" containerName="payload-processing"
Sep 10 00:26:20.197225 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197205    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6349746d-5914-4a80-a580-ec20f45b1fd6" containerName="payload-processing"
Sep 10 00:26:20.197225 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197212    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="604c0c29-4a54-4ed2-9087-a8f627002be2" containerName="revoke-keys"
Sep 10 00:26:20.197225 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197218    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="604c0c29-4a54-4ed2-9087-a8f627002be2" containerName="revoke-keys"
Sep 10 00:26:20.197225 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197226    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c2cdfae-2571-498c-857b-fe94cd2bccd8" containerName="payload-pre-processing"
Sep 10 00:26:20.197225 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197231    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c2cdfae-2571-498c-857b-fe94cd2bccd8" containerName="payload-pre-processing"
Sep 10 00:26:20.197485 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197289    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c2cdfae-2571-498c-857b-fe94cd2bccd8" containerName="payload-pre-processing"
Sep 10 00:26:20.197485 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197299    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="604c0c29-4a54-4ed2-9087-a8f627002be2" containerName="revoke-keys"
Sep 10 00:26:20.197485 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197305    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="14837297-91a9-440a-865f-d949d9c1d740" containerName="main"
Sep 10 00:26:20.197485 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.197312    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6349746d-5914-4a80-a580-ec20f45b1fd6" containerName="payload-processing"
Sep 10 00:26:20.200276 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.200260    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"
Sep 10 00:26:20.202657 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.202636    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:26:20.202821 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.202801    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:26:20.207002 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.206975    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"]
Sep 10 00:26:20.249501 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.249469    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276e01f-3c00-40dd-a762-5dab87884b90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx\" (UID: \"4276e01f-3c00-40dd-a762-5dab87884b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"
Sep 10 00:26:20.350389 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.350360    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276e01f-3c00-40dd-a762-5dab87884b90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx\" (UID: \"4276e01f-3c00-40dd-a762-5dab87884b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"
Sep 10 00:26:20.350938 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.350918    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276e01f-3c00-40dd-a762-5dab87884b90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx\" (UID: \"4276e01f-3c00-40dd-a762-5dab87884b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"
Sep 10 00:26:20.512305 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.512277    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"
Sep 10 00:26:20.779628 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:20.779604    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"]
Sep 10 00:26:20.781724 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:26:20.781698    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4276e01f_3c00_40dd_a762_5dab87884b90.slice/crio-f1cfc72e3b00cf5f5978168eedb903a875e3430bf960fa0a2e2b70210e1bf9ac WatchSource:0}: Error finding container f1cfc72e3b00cf5f5978168eedb903a875e3430bf960fa0a2e2b70210e1bf9ac: Status 404 returned error can't find the container with id f1cfc72e3b00cf5f5978168eedb903a875e3430bf960fa0a2e2b70210e1bf9ac
Sep 10 00:26:21.017950 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:21.017926    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx" event={"ID":"4276e01f-3c00-40dd-a762-5dab87884b90","Type":"ContainerStarted","Data":"f1cfc72e3b00cf5f5978168eedb903a875e3430bf960fa0a2e2b70210e1bf9ac"}
Sep 10 00:26:22.024895 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:22.024861    2556 generic.go:356] "Generic (PLEG): container finished" podID="4276e01f-3c00-40dd-a762-5dab87884b90" containerID="b0875e665a863fee189e16a88b977142fa44dcd1e8f27af034bac36abb2bf5a8" exitCode=0
Sep 10 00:26:22.025276 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:22.024934    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx" event={"ID":"4276e01f-3c00-40dd-a762-5dab87884b90","Type":"ContainerDied","Data":"b0875e665a863fee189e16a88b977142fa44dcd1e8f27af034bac36abb2bf5a8"}
Sep 10 00:26:23.160665 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:23.160641    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"
Sep 10 00:26:23.274744 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:23.274718    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276e01f-3c00-40dd-a762-5dab87884b90-maas-api-service-ca\") pod \"4276e01f-3c00-40dd-a762-5dab87884b90\" (UID: \"4276e01f-3c00-40dd-a762-5dab87884b90\") "
Sep 10 00:26:23.275017 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:23.274996    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4276e01f-3c00-40dd-a762-5dab87884b90-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4276e01f-3c00-40dd-a762-5dab87884b90" (UID: "4276e01f-3c00-40dd-a762-5dab87884b90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:26:23.375773 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:23.375724    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276e01f-3c00-40dd-a762-5dab87884b90-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:26:24.036295 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:24.036274    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"
Sep 10 00:26:24.036444 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:24.036311    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx" event={"ID":"4276e01f-3c00-40dd-a762-5dab87884b90","Type":"ContainerDied","Data":"f1cfc72e3b00cf5f5978168eedb903a875e3430bf960fa0a2e2b70210e1bf9ac"}
Sep 10 00:26:24.036444 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:24.036361    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1cfc72e3b00cf5f5978168eedb903a875e3430bf960fa0a2e2b70210e1bf9ac"
Sep 10 00:26:30.636615 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:30.636582    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"]
Sep 10 00:26:30.637290 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:30.636882    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9" podUID="b3130c5c-ba48-4600-b6f6-d40548e491a8" containerName="payload-processing" containerID="cri-o://2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8" gracePeriod=30
Sep 10 00:26:30.639978 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:30.639950    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"]
Sep 10 00:26:30.640195 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:30.640164    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n" podUID="4e7ab857-48a2-4a7e-aa0d-802783386fe9" containerName="payload-pre-processing" containerID="cri-o://bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d" gracePeriod=30
Sep 10 00:26:30.782289 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:30.782191    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"]
Sep 10 00:26:30.784439 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:30.784415    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c9018-42jxx"]
Sep 10 00:26:30.904548 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:30.904524    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:26:30.940744 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:30.940717    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:26:31.037712 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.037687    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dbpr\" (UniqueName: \"kubernetes.io/projected/b3130c5c-ba48-4600-b6f6-d40548e491a8-kube-api-access-4dbpr\") pod \"b3130c5c-ba48-4600-b6f6-d40548e491a8\" (UID: \"b3130c5c-ba48-4600-b6f6-d40548e491a8\") "
Sep 10 00:26:31.037856 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.037741    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7ab857-48a2-4a7e-aa0d-802783386fe9-plugins-config-volume\") pod \"4e7ab857-48a2-4a7e-aa0d-802783386fe9\" (UID: \"4e7ab857-48a2-4a7e-aa0d-802783386fe9\") "
Sep 10 00:26:31.037856 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.037797    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3130c5c-ba48-4600-b6f6-d40548e491a8-plugins-config-volume\") pod \"b3130c5c-ba48-4600-b6f6-d40548e491a8\" (UID: \"b3130c5c-ba48-4600-b6f6-d40548e491a8\") "
Sep 10 00:26:31.037856 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.037814    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl2lq\" (UniqueName: \"kubernetes.io/projected/4e7ab857-48a2-4a7e-aa0d-802783386fe9-kube-api-access-cl2lq\") pod \"4e7ab857-48a2-4a7e-aa0d-802783386fe9\" (UID: \"4e7ab857-48a2-4a7e-aa0d-802783386fe9\") "
Sep 10 00:26:31.038212 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.038187    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3130c5c-ba48-4600-b6f6-d40548e491a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3130c5c-ba48-4600-b6f6-d40548e491a8" (UID: "b3130c5c-ba48-4600-b6f6-d40548e491a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:26:31.038212 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.038199    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7ab857-48a2-4a7e-aa0d-802783386fe9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e7ab857-48a2-4a7e-aa0d-802783386fe9" (UID: "4e7ab857-48a2-4a7e-aa0d-802783386fe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:26:31.039614 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.039590    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3130c5c-ba48-4600-b6f6-d40548e491a8-kube-api-access-4dbpr" (OuterVolumeSpecName: "kube-api-access-4dbpr") pod "b3130c5c-ba48-4600-b6f6-d40548e491a8" (UID: "b3130c5c-ba48-4600-b6f6-d40548e491a8"). InnerVolumeSpecName "kube-api-access-4dbpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:26:31.039779 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.039763    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7ab857-48a2-4a7e-aa0d-802783386fe9-kube-api-access-cl2lq" (OuterVolumeSpecName: "kube-api-access-cl2lq") pod "4e7ab857-48a2-4a7e-aa0d-802783386fe9" (UID: "4e7ab857-48a2-4a7e-aa0d-802783386fe9"). InnerVolumeSpecName "kube-api-access-cl2lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:26:31.073285 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.073260    2556 generic.go:356] "Generic (PLEG): container finished" podID="4e7ab857-48a2-4a7e-aa0d-802783386fe9" containerID="bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d" exitCode=0
Sep 10 00:26:31.073390 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.073305    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"
Sep 10 00:26:31.073390 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.073347    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n" event={"ID":"4e7ab857-48a2-4a7e-aa0d-802783386fe9","Type":"ContainerDied","Data":"bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d"}
Sep 10 00:26:31.073488 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.073436    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n" event={"ID":"4e7ab857-48a2-4a7e-aa0d-802783386fe9","Type":"ContainerDied","Data":"3d6e184b6ea649c7ae3219840ebd0faec401ede2d4fce3233f69e2fa3ea911b1"}
Sep 10 00:26:31.073488 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.073464    2556 scope.go:117] "RemoveContainer" containerID="bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d"
Sep 10 00:26:31.074707 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.074683    2556 generic.go:356] "Generic (PLEG): container finished" podID="b3130c5c-ba48-4600-b6f6-d40548e491a8" containerID="2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8" exitCode=0
Sep 10 00:26:31.074826 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.074717    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"
Sep 10 00:26:31.074826 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.074761    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9" event={"ID":"b3130c5c-ba48-4600-b6f6-d40548e491a8","Type":"ContainerDied","Data":"2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8"}
Sep 10 00:26:31.074826 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.074795    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9" event={"ID":"b3130c5c-ba48-4600-b6f6-d40548e491a8","Type":"ContainerDied","Data":"2d036f6b8f9f57d0d036f9a5a3eb1b7ed209bd8052e455b55696de75848c1669"}
Sep 10 00:26:31.082124 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.082070    2556 scope.go:117] "RemoveContainer" containerID="bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d"
Sep 10 00:26:31.082408 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:26:31.082381    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d\": container with ID starting with bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d not found: ID does not exist" containerID="bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d"
Sep 10 00:26:31.082494 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.082417    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d"} err="failed to get container status \"bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d\": rpc error: code = NotFound desc = could not find container \"bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d\": container with ID starting with bc7b7398cd602f8664e220e5d616ac97a3d4d5ab8c691165d3954fa9f2900d8d not found: ID does not exist"
Sep 10 00:26:31.082494 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.082442    2556 scope.go:117] "RemoveContainer" containerID="2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8"
Sep 10 00:26:31.090408 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.090389    2556 scope.go:117] "RemoveContainer" containerID="2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8"
Sep 10 00:26:31.090647 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:26:31.090630    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8\": container with ID starting with 2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8 not found: ID does not exist" containerID="2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8"
Sep 10 00:26:31.090683 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.090655    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8"} err="failed to get container status \"2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8\": rpc error: code = NotFound desc = could not find container \"2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8\": container with ID starting with 2377bca7f3bef23437813d0dc73d75ab6c86ef10f4484d11051a7e5e5ebc1ff8 not found: ID does not exist"
Sep 10 00:26:31.096992 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.096974    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"]
Sep 10 00:26:31.100543 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.100520    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c9018-6fcc79d479-jtww9"]
Sep 10 00:26:31.115252 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.115233    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"]
Sep 10 00:26:31.119128 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.119107    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c9018-5b7589b586-mtk5n"]
Sep 10 00:26:31.138806 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.138763    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dbpr\" (UniqueName: \"kubernetes.io/projected/b3130c5c-ba48-4600-b6f6-d40548e491a8-kube-api-access-4dbpr\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:26:31.138806 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.138782    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7ab857-48a2-4a7e-aa0d-802783386fe9-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:26:31.138806 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.138792    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3130c5c-ba48-4600-b6f6-d40548e491a8-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:26:31.138806 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:31.138801    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl2lq\" (UniqueName: \"kubernetes.io/projected/4e7ab857-48a2-4a7e-aa0d-802783386fe9-kube-api-access-cl2lq\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:26:32.267883 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:32.267848    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4276e01f-3c00-40dd-a762-5dab87884b90" path="/var/lib/kubelet/pods/4276e01f-3c00-40dd-a762-5dab87884b90/volumes"
Sep 10 00:26:32.268294 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:32.268275    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e7ab857-48a2-4a7e-aa0d-802783386fe9" path="/var/lib/kubelet/pods/4e7ab857-48a2-4a7e-aa0d-802783386fe9/volumes"
Sep 10 00:26:32.268652 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:32.268637    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3130c5c-ba48-4600-b6f6-d40548e491a8" path="/var/lib/kubelet/pods/b3130c5c-ba48-4600-b6f6-d40548e491a8/volumes"
Sep 10 00:26:37.699309 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.699272    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"]
Sep 10 00:26:37.699898 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.699877    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3130c5c-ba48-4600-b6f6-d40548e491a8" containerName="payload-processing"
Sep 10 00:26:37.699982 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.699901    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3130c5c-ba48-4600-b6f6-d40548e491a8" containerName="payload-processing"
Sep 10 00:26:37.699982 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.699922    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4276e01f-3c00-40dd-a762-5dab87884b90" containerName="revoke-keys"
Sep 10 00:26:37.699982 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.699931    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4276e01f-3c00-40dd-a762-5dab87884b90" containerName="revoke-keys"
Sep 10 00:26:37.699982 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.699959    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7ab857-48a2-4a7e-aa0d-802783386fe9" containerName="payload-pre-processing"
Sep 10 00:26:37.699982 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.699968    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7ab857-48a2-4a7e-aa0d-802783386fe9" containerName="payload-pre-processing"
Sep 10 00:26:37.700255 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.700076    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3130c5c-ba48-4600-b6f6-d40548e491a8" containerName="payload-processing"
Sep 10 00:26:37.700255 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.700093    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7ab857-48a2-4a7e-aa0d-802783386fe9" containerName="payload-pre-processing"
Sep 10 00:26:37.700255 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.700104    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4276e01f-3c00-40dd-a762-5dab87884b90" containerName="revoke-keys"
Sep 10 00:26:37.704622 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.704603    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"
Sep 10 00:26:37.706892 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.706872    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:26:37.707167 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.706876    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:26:37.710560 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.710541    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"]
Sep 10 00:26:37.791637 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.791605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4402ee-1876-4ca5-92ff-ce723fb5da43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp\" (UID: \"3f4402ee-1876-4ca5-92ff-ce723fb5da43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"
Sep 10 00:26:37.892644 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.892621    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4402ee-1876-4ca5-92ff-ce723fb5da43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp\" (UID: \"3f4402ee-1876-4ca5-92ff-ce723fb5da43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"
Sep 10 00:26:37.893241 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:37.893222    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4402ee-1876-4ca5-92ff-ce723fb5da43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp\" (UID: \"3f4402ee-1876-4ca5-92ff-ce723fb5da43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"
Sep 10 00:26:38.017507 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:38.017479    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"
Sep 10 00:26:38.285522 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:38.285498    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"]
Sep 10 00:26:38.287299 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:26:38.287271    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4402ee_1876_4ca5_92ff_ce723fb5da43.slice/crio-0c9c3e73f8bb8caa343d2e53874cfaae03f622394944cd34a300522075efd0a6 WatchSource:0}: Error finding container 0c9c3e73f8bb8caa343d2e53874cfaae03f622394944cd34a300522075efd0a6: Status 404 returned error can't find the container with id 0c9c3e73f8bb8caa343d2e53874cfaae03f622394944cd34a300522075efd0a6
Sep 10 00:26:39.118901 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:39.118866    2556 generic.go:356] "Generic (PLEG): container finished" podID="3f4402ee-1876-4ca5-92ff-ce723fb5da43" containerID="0b949759c14d13a1e7104d76511495c5e5b110d7fc1d3a5ea5cbd1ba39b36a40" exitCode=0
Sep 10 00:26:39.119250 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:39.118924    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp" event={"ID":"3f4402ee-1876-4ca5-92ff-ce723fb5da43","Type":"ContainerDied","Data":"0b949759c14d13a1e7104d76511495c5e5b110d7fc1d3a5ea5cbd1ba39b36a40"}
Sep 10 00:26:39.119250 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:39.118977    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp" event={"ID":"3f4402ee-1876-4ca5-92ff-ce723fb5da43","Type":"ContainerStarted","Data":"0c9c3e73f8bb8caa343d2e53874cfaae03f622394944cd34a300522075efd0a6"}
Sep 10 00:26:40.249537 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:40.249516    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"
Sep 10 00:26:40.411441 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:40.411373    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4402ee-1876-4ca5-92ff-ce723fb5da43-maas-api-service-ca\") pod \"3f4402ee-1876-4ca5-92ff-ce723fb5da43\" (UID: \"3f4402ee-1876-4ca5-92ff-ce723fb5da43\") "
Sep 10 00:26:40.411694 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:40.411674    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4402ee-1876-4ca5-92ff-ce723fb5da43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f4402ee-1876-4ca5-92ff-ce723fb5da43" (UID: "3f4402ee-1876-4ca5-92ff-ce723fb5da43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:26:40.512562 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:40.512530    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4402ee-1876-4ca5-92ff-ce723fb5da43-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:26:41.129879 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:41.129806    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"
Sep 10 00:26:41.129879 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:41.129817    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp" event={"ID":"3f4402ee-1876-4ca5-92ff-ce723fb5da43","Type":"ContainerDied","Data":"0c9c3e73f8bb8caa343d2e53874cfaae03f622394944cd34a300522075efd0a6"}
Sep 10 00:26:41.129879 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:41.129863    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c9c3e73f8bb8caa343d2e53874cfaae03f622394944cd34a300522075efd0a6"
Sep 10 00:26:49.038554 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:49.038514    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr"]
Sep 10 00:26:49.040316 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:49.040288    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1de9a7-29816655-prgwr"]
Sep 10 00:26:49.216637 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:49.216600    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"]
Sep 10 00:26:49.221245 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:49.221219    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1de9a7-tsdpp"]
Sep 10 00:26:50.268680 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:50.268643    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f4402ee-1876-4ca5-92ff-ce723fb5da43" path="/var/lib/kubelet/pods/3f4402ee-1876-4ca5-92ff-ce723fb5da43/volumes"
Sep 10 00:26:50.269094 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:50.268957    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa08f2f2-1184-449c-983f-d26a8921ab0d" path="/var/lib/kubelet/pods/aa08f2f2-1184-449c-983f-d26a8921ab0d/volumes"
Sep 10 00:26:56.183590 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.183557    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"]
Sep 10 00:26:56.183973 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.183966    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4402ee-1876-4ca5-92ff-ce723fb5da43" containerName="revoke-keys"
Sep 10 00:26:56.184021 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.183978    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4402ee-1876-4ca5-92ff-ce723fb5da43" containerName="revoke-keys"
Sep 10 00:26:56.184079 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.184073    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f4402ee-1876-4ca5-92ff-ce723fb5da43" containerName="revoke-keys"
Sep 10 00:26:56.187419 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.187403    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"
Sep 10 00:26:56.189759 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.189737    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 00:26:56.189892 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.189738    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mxc4l\""
Sep 10 00:26:56.193522 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.193500    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"]
Sep 10 00:26:56.338400 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.338367    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcec04dd-d0bf-41f8-98c5-65973e998227-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms\" (UID: \"bcec04dd-d0bf-41f8-98c5-65973e998227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"
Sep 10 00:26:56.438955 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.438889    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcec04dd-d0bf-41f8-98c5-65973e998227-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms\" (UID: \"bcec04dd-d0bf-41f8-98c5-65973e998227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"
Sep 10 00:26:56.439532 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.439512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcec04dd-d0bf-41f8-98c5-65973e998227-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms\" (UID: \"bcec04dd-d0bf-41f8-98c5-65973e998227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"
Sep 10 00:26:56.499947 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.499917    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"
Sep 10 00:26:56.768339 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:56.768313    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"]
Sep 10 00:26:56.770156 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:26:56.770129    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcec04dd_d0bf_41f8_98c5_65973e998227.slice/crio-b7985b6c9799aa8a49de2e26b8a0ffd43f9904989c3ef38d4b0f82174e51420f WatchSource:0}: Error finding container b7985b6c9799aa8a49de2e26b8a0ffd43f9904989c3ef38d4b0f82174e51420f: Status 404 returned error can't find the container with id b7985b6c9799aa8a49de2e26b8a0ffd43f9904989c3ef38d4b0f82174e51420f
Sep 10 00:26:57.221740 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:57.221708    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms" event={"ID":"bcec04dd-d0bf-41f8-98c5-65973e998227","Type":"ContainerStarted","Data":"cfb599cbefe859100a57398df5ca08ddb85f676ace0b4f69668f2ee00eab0eaf"}
Sep 10 00:26:57.222162 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:57.221759    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms" event={"ID":"bcec04dd-d0bf-41f8-98c5-65973e998227","Type":"ContainerStarted","Data":"b7985b6c9799aa8a49de2e26b8a0ffd43f9904989c3ef38d4b0f82174e51420f"}
Sep 10 00:26:57.235196 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:57.235153    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms" podStartSLOduration=1.235138168 podStartE2EDuration="1.235138168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:26:57.234444854 +0000 UTC m=+2297.485580900" watchObservedRunningTime="2026-09-10 00:26:57.235138168 +0000 UTC m=+2297.486274210"
Sep 10 00:26:58.229337 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:58.229298    2556 generic.go:356] "Generic (PLEG): container finished" podID="bcec04dd-d0bf-41f8-98c5-65973e998227" containerID="cfb599cbefe859100a57398df5ca08ddb85f676ace0b4f69668f2ee00eab0eaf" exitCode=0
Sep 10 00:26:58.229713 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:58.229341    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms" event={"ID":"bcec04dd-d0bf-41f8-98c5-65973e998227","Type":"ContainerDied","Data":"cfb599cbefe859100a57398df5ca08ddb85f676ace0b4f69668f2ee00eab0eaf"}
Sep 10 00:26:59.363345 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:59.363321    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"
Sep 10 00:26:59.460497 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:59.460471    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcec04dd-d0bf-41f8-98c5-65973e998227-maas-api-service-ca\") pod \"bcec04dd-d0bf-41f8-98c5-65973e998227\" (UID: \"bcec04dd-d0bf-41f8-98c5-65973e998227\") "
Sep 10 00:26:59.460784 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:59.460763    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcec04dd-d0bf-41f8-98c5-65973e998227-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bcec04dd-d0bf-41f8-98c5-65973e998227" (UID: "bcec04dd-d0bf-41f8-98c5-65973e998227"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:26:59.561152 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:26:59.561094    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcec04dd-d0bf-41f8-98c5-65973e998227-maas-api-service-ca\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:27:00.242719 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:00.242692    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"
Sep 10 00:27:00.242884 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:00.242698    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms" event={"ID":"bcec04dd-d0bf-41f8-98c5-65973e998227","Type":"ContainerDied","Data":"b7985b6c9799aa8a49de2e26b8a0ffd43f9904989c3ef38d4b0f82174e51420f"}
Sep 10 00:27:00.242884 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:00.242801    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7985b6c9799aa8a49de2e26b8a0ffd43f9904989c3ef38d4b0f82174e51420f"
Sep 10 00:27:06.579847 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:06.579811    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq"]
Sep 10 00:27:06.581741 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:06.581714    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b1baac-29816655-5rsmq"]
Sep 10 00:27:06.608874 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:06.608844    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"]
Sep 10 00:27:06.609160 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:06.609133    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l" podUID="f18a10ba-2fbc-4aad-98a8-1423938ee772" containerName="payload-pre-processing" containerID="cri-o://410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4" gracePeriod=30
Sep 10 00:27:06.755220 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:06.755157    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"]
Sep 10 00:27:06.756879 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:06.756853    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b1baac-qh4ms"]
Sep 10 00:27:06.855084 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:06.855046    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:27:06.919709 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:06.919685    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18a10ba-2fbc-4aad-98a8-1423938ee772-plugins-config-volume\") pod \"f18a10ba-2fbc-4aad-98a8-1423938ee772\" (UID: \"f18a10ba-2fbc-4aad-98a8-1423938ee772\") "
Sep 10 00:27:06.919846 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:06.919736    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmjfl\" (UniqueName: \"kubernetes.io/projected/f18a10ba-2fbc-4aad-98a8-1423938ee772-kube-api-access-pmjfl\") pod \"f18a10ba-2fbc-4aad-98a8-1423938ee772\" (UID: \"f18a10ba-2fbc-4aad-98a8-1423938ee772\") "
Sep 10 00:27:06.920027 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:06.920004    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18a10ba-2fbc-4aad-98a8-1423938ee772-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f18a10ba-2fbc-4aad-98a8-1423938ee772" (UID: "f18a10ba-2fbc-4aad-98a8-1423938ee772"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 00:27:06.921592 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:06.921573    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18a10ba-2fbc-4aad-98a8-1423938ee772-kube-api-access-pmjfl" (OuterVolumeSpecName: "kube-api-access-pmjfl") pod "f18a10ba-2fbc-4aad-98a8-1423938ee772" (UID: "f18a10ba-2fbc-4aad-98a8-1423938ee772"). InnerVolumeSpecName "kube-api-access-pmjfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:27:07.020923 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:07.020892    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f18a10ba-2fbc-4aad-98a8-1423938ee772-plugins-config-volume\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:27:07.020923 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:07.020920    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmjfl\" (UniqueName: \"kubernetes.io/projected/f18a10ba-2fbc-4aad-98a8-1423938ee772-kube-api-access-pmjfl\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:27:07.280110 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:07.280078    2556 generic.go:356] "Generic (PLEG): container finished" podID="f18a10ba-2fbc-4aad-98a8-1423938ee772" containerID="410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4" exitCode=0
Sep 10 00:27:07.280241 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:07.280120    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l" event={"ID":"f18a10ba-2fbc-4aad-98a8-1423938ee772","Type":"ContainerDied","Data":"410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4"}
Sep 10 00:27:07.280241 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:07.280139    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"
Sep 10 00:27:07.280241 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:07.280165    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l" event={"ID":"f18a10ba-2fbc-4aad-98a8-1423938ee772","Type":"ContainerDied","Data":"b7e79f9d5bc958f9608a1e5bc45bf5157db61ce9b54e0e64724eaa490ebe876c"}
Sep 10 00:27:07.280241 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:07.280195    2556 scope.go:117] "RemoveContainer" containerID="410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4"
Sep 10 00:27:07.289469 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:07.289453    2556 scope.go:117] "RemoveContainer" containerID="410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4"
Sep 10 00:27:07.289757 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:27:07.289733    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4\": container with ID starting with 410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4 not found: ID does not exist" containerID="410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4"
Sep 10 00:27:07.289853 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:07.289761    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4"} err="failed to get container status \"410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4\": rpc error: code = NotFound desc = could not find container \"410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4\": container with ID starting with 410f5bd350736d7d979c985edaab6600d3976d4739c9782879980a1ed39bd7d4 not found: ID does not exist"
Sep 10 00:27:07.299681 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:07.299660    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"]
Sep 10 00:27:07.303441 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:07.303420    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-b1baac-5497cdb49d-8bj2l"]
Sep 10 00:27:08.267822 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:08.267789    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ba1b84-15ab-46c8-9b01-b790ae35cc90" path="/var/lib/kubelet/pods/31ba1b84-15ab-46c8-9b01-b790ae35cc90/volumes"
Sep 10 00:27:08.268201 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:08.268129    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcec04dd-d0bf-41f8-98c5-65973e998227" path="/var/lib/kubelet/pods/bcec04dd-d0bf-41f8-98c5-65973e998227/volumes"
Sep 10 00:27:08.268492 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:08.268476    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f18a10ba-2fbc-4aad-98a8-1423938ee772" path="/var/lib/kubelet/pods/f18a10ba-2fbc-4aad-98a8-1423938ee772/volumes"
Sep 10 00:27:16.019513 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.019484    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-wmmzm"]
Sep 10 00:27:16.020139 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.019672    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f8584cb86-wmmzm" podUID="4d738e4c-0a56-40dd-af76-dbfac13e2dae" containerName="manager" containerID="cri-o://3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9" gracePeriod=30
Sep 10 00:27:16.263946 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.263926    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:27:16.303458 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.303382    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbw4t\" (UniqueName: \"kubernetes.io/projected/4d738e4c-0a56-40dd-af76-dbfac13e2dae-kube-api-access-kbw4t\") pod \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") "
Sep 10 00:27:16.303607 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.303471    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-metrics-certs\") pod \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") "
Sep 10 00:27:16.303607 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.303510    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-webhook-cert\") pod \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\" (UID: \"4d738e4c-0a56-40dd-af76-dbfac13e2dae\") "
Sep 10 00:27:16.305649 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.305621    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d738e4c-0a56-40dd-af76-dbfac13e2dae-kube-api-access-kbw4t" (OuterVolumeSpecName: "kube-api-access-kbw4t") pod "4d738e4c-0a56-40dd-af76-dbfac13e2dae" (UID: "4d738e4c-0a56-40dd-af76-dbfac13e2dae"). InnerVolumeSpecName "kube-api-access-kbw4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:27:16.305978 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.305951    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4d738e4c-0a56-40dd-af76-dbfac13e2dae" (UID: "4d738e4c-0a56-40dd-af76-dbfac13e2dae"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:27:16.306082 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.306003    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4d738e4c-0a56-40dd-af76-dbfac13e2dae" (UID: "4d738e4c-0a56-40dd-af76-dbfac13e2dae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:27:16.327460 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.327430    2556 generic.go:356] "Generic (PLEG): container finished" podID="4d738e4c-0a56-40dd-af76-dbfac13e2dae" containerID="3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9" exitCode=0
Sep 10 00:27:16.327564 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.327476    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f8584cb86-wmmzm"
Sep 10 00:27:16.327564 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.327530    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f8584cb86-wmmzm" event={"ID":"4d738e4c-0a56-40dd-af76-dbfac13e2dae","Type":"ContainerDied","Data":"3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9"}
Sep 10 00:27:16.327648 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.327576    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f8584cb86-wmmzm" event={"ID":"4d738e4c-0a56-40dd-af76-dbfac13e2dae","Type":"ContainerDied","Data":"e207b89f4cdb671de5f006135d37d3093b858a6b7fc518e81bbee5567955faf3"}
Sep 10 00:27:16.327648 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.327597    2556 scope.go:117] "RemoveContainer" containerID="3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9"
Sep 10 00:27:16.338379 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.338355    2556 scope.go:117] "RemoveContainer" containerID="3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9"
Sep 10 00:27:16.338668 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:27:16.338646    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9\": container with ID starting with 3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9 not found: ID does not exist" containerID="3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9"
Sep 10 00:27:16.338774 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.338676    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9"} err="failed to get container status \"3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9\": rpc error: code = NotFound desc = could not find container \"3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9\": container with ID starting with 3bdec7f65916dd74208f7e43d33444203e700890a03b314c2623552501416be9 not found: ID does not exist"
Sep 10 00:27:16.349947 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.349925    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-wmmzm"]
Sep 10 00:27:16.353334 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.353313    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-wmmzm"]
Sep 10 00:27:16.404865 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.404845    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbw4t\" (UniqueName: \"kubernetes.io/projected/4d738e4c-0a56-40dd-af76-dbfac13e2dae-kube-api-access-kbw4t\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:27:16.404955 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.404870    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-metrics-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:27:16.404955 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:16.404882    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d738e4c-0a56-40dd-af76-dbfac13e2dae-webhook-cert\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:27:18.268488 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:18.268451    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d738e4c-0a56-40dd-af76-dbfac13e2dae" path="/var/lib/kubelet/pods/4d738e4c-0a56-40dd-af76-dbfac13e2dae/volumes"
Sep 10 00:27:40.846823 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:40.846781    2556 scope.go:117] "RemoveContainer" containerID="7dc294a72b75c729302de837f8f589893b12eb6af8aba4aed7bf847945cd46a9"
Sep 10 00:27:40.856547 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:27:40.856505    2556 scope.go:117] "RemoveContainer" containerID="b0ff77ec7d1688f56b902a367dee5ee47b870f9a7d9166cf4985ca29e64e15d4"
Sep 10 00:28:17.145727 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.145686    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f8584cb86-6hkwv"]
Sep 10 00:28:17.146202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.146099    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d738e4c-0a56-40dd-af76-dbfac13e2dae" containerName="manager"
Sep 10 00:28:17.146202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.146111    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d738e4c-0a56-40dd-af76-dbfac13e2dae" containerName="manager"
Sep 10 00:28:17.146202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.146119    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18a10ba-2fbc-4aad-98a8-1423938ee772" containerName="payload-pre-processing"
Sep 10 00:28:17.146202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.146124    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18a10ba-2fbc-4aad-98a8-1423938ee772" containerName="payload-pre-processing"
Sep 10 00:28:17.146202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.146130    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcec04dd-d0bf-41f8-98c5-65973e998227" containerName="revoke-keys"
Sep 10 00:28:17.146202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.146136    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcec04dd-d0bf-41f8-98c5-65973e998227" containerName="revoke-keys"
Sep 10 00:28:17.146202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.146199    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d738e4c-0a56-40dd-af76-dbfac13e2dae" containerName="manager"
Sep 10 00:28:17.146436 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.146210    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18a10ba-2fbc-4aad-98a8-1423938ee772" containerName="payload-pre-processing"
Sep 10 00:28:17.146436 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.146217    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcec04dd-d0bf-41f8-98c5-65973e998227" containerName="revoke-keys"
Sep 10 00:28:17.149210 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.149192    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:17.152421 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.152366    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 00:28:17.152558 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.152509    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 00:28:17.152558 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.152542    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cx96n\""
Sep 10 00:28:17.152690 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.152509    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 00:28:17.158943 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.158923    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-6hkwv"]
Sep 10 00:28:17.285363 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.285335    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-metrics-certs\") pod \"maas-controller-6f8584cb86-6hkwv\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") " pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:17.285503 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.285381    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-webhook-cert\") pod \"maas-controller-6f8584cb86-6hkwv\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") " pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:17.285503 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.285416    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw2m4\" (UniqueName: \"kubernetes.io/projected/f3526171-d3b6-4b01-a803-3dae9ecc45df-kube-api-access-nw2m4\") pod \"maas-controller-6f8584cb86-6hkwv\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") " pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:17.385899 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.385876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-webhook-cert\") pod \"maas-controller-6f8584cb86-6hkwv\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") " pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:17.386047 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.385912    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw2m4\" (UniqueName: \"kubernetes.io/projected/f3526171-d3b6-4b01-a803-3dae9ecc45df-kube-api-access-nw2m4\") pod \"maas-controller-6f8584cb86-6hkwv\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") " pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:17.386047 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.385974    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-metrics-certs\") pod \"maas-controller-6f8584cb86-6hkwv\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") " pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:17.388307 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.388284    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-webhook-cert\") pod \"maas-controller-6f8584cb86-6hkwv\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") " pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:17.388413 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.388325    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-metrics-certs\") pod \"maas-controller-6f8584cb86-6hkwv\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") " pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:17.392871 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.392849    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw2m4\" (UniqueName: \"kubernetes.io/projected/f3526171-d3b6-4b01-a803-3dae9ecc45df-kube-api-access-nw2m4\") pod \"maas-controller-6f8584cb86-6hkwv\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") " pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:17.460837 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.460813    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:17.735950 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.735926    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-6hkwv"]
Sep 10 00:28:17.738105 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:28:17.738081    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3526171_d3b6_4b01_a803_3dae9ecc45df.slice/crio-7d5cdb0957e2c61ab511b31b8d83e4f03e74cd605f707a36ee4f09949852c84e WatchSource:0}: Error finding container 7d5cdb0957e2c61ab511b31b8d83e4f03e74cd605f707a36ee4f09949852c84e: Status 404 returned error can't find the container with id 7d5cdb0957e2c61ab511b31b8d83e4f03e74cd605f707a36ee4f09949852c84e
Sep 10 00:28:17.739316 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:17.739295    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 00:28:18.661879 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:18.661842    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f8584cb86-6hkwv" event={"ID":"f3526171-d3b6-4b01-a803-3dae9ecc45df","Type":"ContainerStarted","Data":"1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1"}
Sep 10 00:28:18.662291 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:18.661895    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f8584cb86-6hkwv" event={"ID":"f3526171-d3b6-4b01-a803-3dae9ecc45df","Type":"ContainerStarted","Data":"7d5cdb0957e2c61ab511b31b8d83e4f03e74cd605f707a36ee4f09949852c84e"}
Sep 10 00:28:18.662291 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:18.661944    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:18.680083 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:18.680025    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f8584cb86-6hkwv" podStartSLOduration=0.96846018 podStartE2EDuration="1.680013437s" totalImagesPullingTime="711.553257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:28:18.677939977 +0000 UTC m=+2378.929076040" watchObservedRunningTime="2026-09-10 00:28:18.680013437 +0000 UTC m=+2378.931149479"
Sep 10 00:28:29.669273 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:29.669237    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:39.002313 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.002277    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-6hkwv"]
Sep 10 00:28:39.002831 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.002464    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f8584cb86-6hkwv" podUID="f3526171-d3b6-4b01-a803-3dae9ecc45df" containerName="manager" containerID="cri-o://1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1" gracePeriod=30
Sep 10 00:28:39.249713 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.249687    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:39.370097 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.370008    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-webhook-cert\") pod \"f3526171-d3b6-4b01-a803-3dae9ecc45df\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") "
Sep 10 00:28:39.370097 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.370050    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw2m4\" (UniqueName: \"kubernetes.io/projected/f3526171-d3b6-4b01-a803-3dae9ecc45df-kube-api-access-nw2m4\") pod \"f3526171-d3b6-4b01-a803-3dae9ecc45df\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") "
Sep 10 00:28:39.370097 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.370086    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-metrics-certs\") pod \"f3526171-d3b6-4b01-a803-3dae9ecc45df\" (UID: \"f3526171-d3b6-4b01-a803-3dae9ecc45df\") "
Sep 10 00:28:39.372088 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.372044    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f3526171-d3b6-4b01-a803-3dae9ecc45df" (UID: "f3526171-d3b6-4b01-a803-3dae9ecc45df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:28:39.372664 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.372641    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3526171-d3b6-4b01-a803-3dae9ecc45df-kube-api-access-nw2m4" (OuterVolumeSpecName: "kube-api-access-nw2m4") pod "f3526171-d3b6-4b01-a803-3dae9ecc45df" (UID: "f3526171-d3b6-4b01-a803-3dae9ecc45df"). InnerVolumeSpecName "kube-api-access-nw2m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 00:28:39.372754 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.372647    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f3526171-d3b6-4b01-a803-3dae9ecc45df" (UID: "f3526171-d3b6-4b01-a803-3dae9ecc45df"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 00:28:39.471248 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.471222    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-webhook-cert\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:28:39.471248 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.471244    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw2m4\" (UniqueName: \"kubernetes.io/projected/f3526171-d3b6-4b01-a803-3dae9ecc45df-kube-api-access-nw2m4\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:28:39.471391 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.471253    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3526171-d3b6-4b01-a803-3dae9ecc45df-metrics-certs\") on node \"ip-10-0-139-211.ec2.internal\" DevicePath \"\""
Sep 10 00:28:39.775078 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.775031    2556 generic.go:356] "Generic (PLEG): container finished" podID="f3526171-d3b6-4b01-a803-3dae9ecc45df" containerID="1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1" exitCode=0
Sep 10 00:28:39.775235 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.775089    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f8584cb86-6hkwv"
Sep 10 00:28:39.775235 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.775116    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f8584cb86-6hkwv" event={"ID":"f3526171-d3b6-4b01-a803-3dae9ecc45df","Type":"ContainerDied","Data":"1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1"}
Sep 10 00:28:39.775235 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.775175    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f8584cb86-6hkwv" event={"ID":"f3526171-d3b6-4b01-a803-3dae9ecc45df","Type":"ContainerDied","Data":"7d5cdb0957e2c61ab511b31b8d83e4f03e74cd605f707a36ee4f09949852c84e"}
Sep 10 00:28:39.775235 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.775200    2556 scope.go:117] "RemoveContainer" containerID="1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1"
Sep 10 00:28:39.785149 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.785129    2556 scope.go:117] "RemoveContainer" containerID="1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1"
Sep 10 00:28:39.785492 ip-10-0-139-211 kubenswrapper[2556]: E0910 00:28:39.785472    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1\": container with ID starting with 1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1 not found: ID does not exist" containerID="1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1"
Sep 10 00:28:39.785558 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.785505    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1"} err="failed to get container status \"1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1\": rpc error: code = NotFound desc = could not find container \"1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1\": container with ID starting with 1f1e339adaa866b97733cf16371c25c9e284b8aee2efa18cb87cc9af0fb717d1 not found: ID does not exist"
Sep 10 00:28:39.795720 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.795696    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-6hkwv"]
Sep 10 00:28:39.797444 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:39.797424    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-6hkwv"]
Sep 10 00:28:40.268772 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.268745    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3526171-d3b6-4b01-a803-3dae9ecc45df" path="/var/lib/kubelet/pods/f3526171-d3b6-4b01-a803-3dae9ecc45df/volumes"
Sep 10 00:28:40.563721 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.563644    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f8584cb86-2sls6"]
Sep 10 00:28:40.564081 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.564065    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3526171-d3b6-4b01-a803-3dae9ecc45df" containerName="manager"
Sep 10 00:28:40.564081 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.564081    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3526171-d3b6-4b01-a803-3dae9ecc45df" containerName="manager"
Sep 10 00:28:40.564170 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.564141    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3526171-d3b6-4b01-a803-3dae9ecc45df" containerName="manager"
Sep 10 00:28:40.568797 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.568777    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:40.571352 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.571328    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 00:28:40.571459 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.571378    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 00:28:40.571885 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.571860    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 00:28:40.572002 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.571941    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cx96n\""
Sep 10 00:28:40.577696 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.577671    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-2sls6"]
Sep 10 00:28:40.681561 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.681537    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee3f0da-1f3d-46b9-9194-e6424233972e-metrics-certs\") pod \"maas-controller-6f8584cb86-2sls6\" (UID: \"3ee3f0da-1f3d-46b9-9194-e6424233972e\") " pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:40.681675 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.681617    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghkfd\" (UniqueName: \"kubernetes.io/projected/3ee3f0da-1f3d-46b9-9194-e6424233972e-kube-api-access-ghkfd\") pod \"maas-controller-6f8584cb86-2sls6\" (UID: \"3ee3f0da-1f3d-46b9-9194-e6424233972e\") " pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:40.681675 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.681671    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ee3f0da-1f3d-46b9-9194-e6424233972e-webhook-cert\") pod \"maas-controller-6f8584cb86-2sls6\" (UID: \"3ee3f0da-1f3d-46b9-9194-e6424233972e\") " pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:40.783066 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.783037    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee3f0da-1f3d-46b9-9194-e6424233972e-metrics-certs\") pod \"maas-controller-6f8584cb86-2sls6\" (UID: \"3ee3f0da-1f3d-46b9-9194-e6424233972e\") " pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:40.783209 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.783111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghkfd\" (UniqueName: \"kubernetes.io/projected/3ee3f0da-1f3d-46b9-9194-e6424233972e-kube-api-access-ghkfd\") pod \"maas-controller-6f8584cb86-2sls6\" (UID: \"3ee3f0da-1f3d-46b9-9194-e6424233972e\") " pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:40.783209 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.783144    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ee3f0da-1f3d-46b9-9194-e6424233972e-webhook-cert\") pod \"maas-controller-6f8584cb86-2sls6\" (UID: \"3ee3f0da-1f3d-46b9-9194-e6424233972e\") " pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:40.785496 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.785475    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ee3f0da-1f3d-46b9-9194-e6424233972e-webhook-cert\") pod \"maas-controller-6f8584cb86-2sls6\" (UID: \"3ee3f0da-1f3d-46b9-9194-e6424233972e\") " pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:40.785636 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.785613    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ee3f0da-1f3d-46b9-9194-e6424233972e-metrics-certs\") pod \"maas-controller-6f8584cb86-2sls6\" (UID: \"3ee3f0da-1f3d-46b9-9194-e6424233972e\") " pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:40.789859 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.789843    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghkfd\" (UniqueName: \"kubernetes.io/projected/3ee3f0da-1f3d-46b9-9194-e6424233972e-kube-api-access-ghkfd\") pod \"maas-controller-6f8584cb86-2sls6\" (UID: \"3ee3f0da-1f3d-46b9-9194-e6424233972e\") " pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:40.880914 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:40.880855    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:41.149537 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:41.149513    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f8584cb86-2sls6"]
Sep 10 00:28:41.154092 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:28:41.154066    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee3f0da_1f3d_46b9_9194_e6424233972e.slice/crio-dd528dbc41e783b2c4f915304cc25b71ce16c30900a1d3cfb7d176b576c487f2 WatchSource:0}: Error finding container dd528dbc41e783b2c4f915304cc25b71ce16c30900a1d3cfb7d176b576c487f2: Status 404 returned error can't find the container with id dd528dbc41e783b2c4f915304cc25b71ce16c30900a1d3cfb7d176b576c487f2
Sep 10 00:28:41.787369 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:41.787327    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f8584cb86-2sls6" event={"ID":"3ee3f0da-1f3d-46b9-9194-e6424233972e","Type":"ContainerStarted","Data":"5b4192a7429ec1c16752dad3fd6c6df86f1950180365b63498e143cf0df0f0d6"}
Sep 10 00:28:41.787799 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:41.787397    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f8584cb86-2sls6" event={"ID":"3ee3f0da-1f3d-46b9-9194-e6424233972e","Type":"ContainerStarted","Data":"dd528dbc41e783b2c4f915304cc25b71ce16c30900a1d3cfb7d176b576c487f2"}
Sep 10 00:28:41.787799 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:41.787427    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:28:41.803090 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:41.803030    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f8584cb86-2sls6" podStartSLOduration=1.390187761 podStartE2EDuration="1.803014597s" totalImagesPullingTime="412.826836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:28:41.801810934 +0000 UTC m=+2402.052946979" watchObservedRunningTime="2026-09-10 00:28:41.803014597 +0000 UTC m=+2402.054150639"
Sep 10 00:28:52.794828 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:28:52.794793    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f8584cb86-2sls6"
Sep 10 00:31:40.917394 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:31:40.917357    2556 scope.go:117] "RemoveContainer" containerID="b620445eb8d3893db770e143f6d2928eac1e67e6ee06dd17a9db6def900a337b"
Sep 10 00:31:40.932282 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:31:40.932263    2556 scope.go:117] "RemoveContainer" containerID="3fcdfb91b3f68a3b0dc50e2e75355c3de40ffe5c9740440bc285057b5de3473a"
Sep 10 00:32:40.959225 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:32:40.959193    2556 scope.go:117] "RemoveContainer" containerID="0b949759c14d13a1e7104d76511495c5e5b110d7fc1d3a5ea5cbd1ba39b36a40"
Sep 10 00:32:40.967703 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:32:40.967684    2556 scope.go:117] "RemoveContainer" containerID="b0875e665a863fee189e16a88b977142fa44dcd1e8f27af034bac36abb2bf5a8"
Sep 10 00:33:17.012135 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:17.012102    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-658dbc9d99-t9lnr_4675932e-4a62-4127-a44d-d2d9d2bbc8f3/authorino/0.log"
Sep 10 00:33:22.721949 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:22.721917    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f8584cb86-2sls6_3ee3f0da-1f3d-46b9-9194-e6424233972e/manager/0.log"
Sep 10 00:33:22.960617 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:22.960586    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cz4kd_54466006-34a1-40ad-be74-b22d9ab53140/manager/0.log"
Sep 10 00:33:23.313583 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:23.313559    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cfd54796-hhlph_b105f9d5-9554-4055-9eb8-582aafbc0298/copy-manifests/0.log"
Sep 10 00:33:23.319606 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:23.319581    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cfd54796-hhlph_b105f9d5-9554-4055-9eb8-582aafbc0298/manager/0.log"
Sep 10 00:33:24.876259 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:24.876227    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-658dbc9d99-t9lnr_4675932e-4a62-4127-a44d-d2d9d2bbc8f3/authorino/0.log"
Sep 10 00:33:24.989703 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:24.989674    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-fbw9v_97caa56d-ac6b-4229-8ca6-bcde14491e2e/manager/0.log"
Sep 10 00:33:26.124552 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:26.124523    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fbcb8d956-mkkn8_485f4554-94b3-4e40-ab8c-aed5f078bb14/kube-auth-proxy/0.log"
Sep 10 00:33:26.461546 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:26.461523    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-f5g58_a5f0c922-b35c-4fe8-8c67-2365c774e22f/payload-pre-processing/0.log"
Sep 10 00:33:27.154920 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:27.154886    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-hb9d2_498f2657-87b5-4e85-ac1b-464cacbaa819/main/0.log"
Sep 10 00:33:27.261328 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:27.261304    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-m8vhs_1e231bfd-72e2-46be-81c6-8ebc86d0c0a0/main/0.log"
Sep 10 00:33:27.494808 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:27.494782    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhd4hj_899caf63-d38d-4856-928e-f4f790c44bc6/main/0.log"
Sep 10 00:33:27.602387 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:27.602364    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-wb9s4_e9571ce8-a6ad-4d01-8744-439040e11b6c/main/0.log"
Sep 10 00:33:33.177502 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.177421    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2hczq/must-gather-m7lhg"]
Sep 10 00:33:33.181377 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.181340    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hczq/must-gather-m7lhg"
Sep 10 00:33:33.183699 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.183675    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2hczq\"/\"default-dockercfg-r77nj\""
Sep 10 00:33:33.184576 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.184562    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hczq\"/\"openshift-service-ca.crt\""
Sep 10 00:33:33.184656 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.184568    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2hczq\"/\"kube-root-ca.crt\""
Sep 10 00:33:33.189437 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.189411    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hczq/must-gather-m7lhg"]
Sep 10 00:33:33.358644 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.358619    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b5t7\" (UniqueName: \"kubernetes.io/projected/7021e4e0-b0d5-4b8f-b44b-3eb38196c68c-kube-api-access-5b5t7\") pod \"must-gather-m7lhg\" (UID: \"7021e4e0-b0d5-4b8f-b44b-3eb38196c68c\") " pod="openshift-must-gather-2hczq/must-gather-m7lhg"
Sep 10 00:33:33.358772 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.358667    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7021e4e0-b0d5-4b8f-b44b-3eb38196c68c-must-gather-output\") pod \"must-gather-m7lhg\" (UID: \"7021e4e0-b0d5-4b8f-b44b-3eb38196c68c\") " pod="openshift-must-gather-2hczq/must-gather-m7lhg"
Sep 10 00:33:33.459658 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.459636    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b5t7\" (UniqueName: \"kubernetes.io/projected/7021e4e0-b0d5-4b8f-b44b-3eb38196c68c-kube-api-access-5b5t7\") pod \"must-gather-m7lhg\" (UID: \"7021e4e0-b0d5-4b8f-b44b-3eb38196c68c\") " pod="openshift-must-gather-2hczq/must-gather-m7lhg"
Sep 10 00:33:33.459775 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.459685    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7021e4e0-b0d5-4b8f-b44b-3eb38196c68c-must-gather-output\") pod \"must-gather-m7lhg\" (UID: \"7021e4e0-b0d5-4b8f-b44b-3eb38196c68c\") " pod="openshift-must-gather-2hczq/must-gather-m7lhg"
Sep 10 00:33:33.460033 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.460014    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7021e4e0-b0d5-4b8f-b44b-3eb38196c68c-must-gather-output\") pod \"must-gather-m7lhg\" (UID: \"7021e4e0-b0d5-4b8f-b44b-3eb38196c68c\") " pod="openshift-must-gather-2hczq/must-gather-m7lhg"
Sep 10 00:33:33.473724 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.473663    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b5t7\" (UniqueName: \"kubernetes.io/projected/7021e4e0-b0d5-4b8f-b44b-3eb38196c68c-kube-api-access-5b5t7\") pod \"must-gather-m7lhg\" (UID: \"7021e4e0-b0d5-4b8f-b44b-3eb38196c68c\") " pod="openshift-must-gather-2hczq/must-gather-m7lhg"
Sep 10 00:33:33.490520 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.490497    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hczq/must-gather-m7lhg"
Sep 10 00:33:33.759079 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.759039    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hczq/must-gather-m7lhg"]
Sep 10 00:33:33.761244 ip-10-0-139-211 kubenswrapper[2556]: W0910 00:33:33.761218    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7021e4e0_b0d5_4b8f_b44b_3eb38196c68c.slice/crio-6aab04fddf53bfc494193a38c80dbe0316d4acd1d3af0b1b086f7bbc7cf6d98f WatchSource:0}: Error finding container 6aab04fddf53bfc494193a38c80dbe0316d4acd1d3af0b1b086f7bbc7cf6d98f: Status 404 returned error can't find the container with id 6aab04fddf53bfc494193a38c80dbe0316d4acd1d3af0b1b086f7bbc7cf6d98f
Sep 10 00:33:33.763550 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:33.763534    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 00:33:34.285410 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:34.285350    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hczq/must-gather-m7lhg" event={"ID":"7021e4e0-b0d5-4b8f-b44b-3eb38196c68c","Type":"ContainerStarted","Data":"6aab04fddf53bfc494193a38c80dbe0316d4acd1d3af0b1b086f7bbc7cf6d98f"}
Sep 10 00:33:36.300104 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:36.300033    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hczq/must-gather-m7lhg" event={"ID":"7021e4e0-b0d5-4b8f-b44b-3eb38196c68c","Type":"ContainerStarted","Data":"cde3896ce6275571eabad39175ac516e0dd1965cd87dca3e1ad0d9d362aa696e"}
Sep 10 00:33:36.300581 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:36.300122    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hczq/must-gather-m7lhg" event={"ID":"7021e4e0-b0d5-4b8f-b44b-3eb38196c68c","Type":"ContainerStarted","Data":"7c9286ab2591ba30817b56367eeb14e7729a876e71b4f5c36dfab5c0767721d2"}
Sep 10 00:33:36.314977 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:36.314917    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2hczq/must-gather-m7lhg" podStartSLOduration=1.772510328 podStartE2EDuration="3.314881595s" totalImagesPullingTime="1.542371267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 00:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:33:36.313522637 +0000 UTC m=+2696.564658681" watchObservedRunningTime="2026-09-10 00:33:36.314881595 +0000 UTC m=+2696.566017640"
Sep 10 00:33:36.813327 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:36.813288    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kg65s_ab3c838b-f89a-4dd8-8318-705775e5bb2a/global-pull-secret-syncer/0.log"
Sep 10 00:33:36.897494 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:36.897464    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7nmdr_d89fc94d-6596-40b6-8565-5bf0277d14da/konnectivity-agent/0.log"
Sep 10 00:33:36.991466 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:36.991435    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-211.ec2.internal_44b8b5997f7574778e566a518d72e361/haproxy/0.log"
Sep 10 00:33:40.992535 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:40.992494    2556 scope.go:117] "RemoveContainer" containerID="cfb599cbefe859100a57398df5ca08ddb85f676ace0b4f69668f2ee00eab0eaf"
Sep 10 00:33:41.966012 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:41.965980    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-658dbc9d99-t9lnr_4675932e-4a62-4127-a44d-d2d9d2bbc8f3/authorino/0.log"
Sep 10 00:33:42.001019 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:42.000990    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-fbw9v_97caa56d-ac6b-4229-8ca6-bcde14491e2e/manager/0.log"
Sep 10 00:33:43.584113 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:43.584049    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_624a9187-3dcc-419b-a663-91159b9879ef/alertmanager/0.log"
Sep 10 00:33:43.604628 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:43.604573    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_624a9187-3dcc-419b-a663-91159b9879ef/config-reloader/0.log"
Sep 10 00:33:43.624545 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:43.624509    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_624a9187-3dcc-419b-a663-91159b9879ef/kube-rbac-proxy-web/0.log"
Sep 10 00:33:43.648768 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:43.648735    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_624a9187-3dcc-419b-a663-91159b9879ef/kube-rbac-proxy/0.log"
Sep 10 00:33:43.669671 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:43.669641    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_624a9187-3dcc-419b-a663-91159b9879ef/kube-rbac-proxy-metric/0.log"
Sep 10 00:33:43.690048 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:43.690017    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_624a9187-3dcc-419b-a663-91159b9879ef/prom-label-proxy/0.log"
Sep 10 00:33:43.710925 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:43.710899    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_624a9187-3dcc-419b-a663-91159b9879ef/init-config-reloader/0.log"
Sep 10 00:33:43.753162 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:43.753126    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-m6xqq_012fd72d-5acd-4896-9a66-ad26dbdc2a4e/cluster-monitoring-operator/0.log"
Sep 10 00:33:43.963979 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:43.963948    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vlxfk_8389ea18-0b63-4ccf-b5ec-2e24b99b35fe/node-exporter/0.log"
Sep 10 00:33:43.984028 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:43.984004    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vlxfk_8389ea18-0b63-4ccf-b5ec-2e24b99b35fe/kube-rbac-proxy/0.log"
Sep 10 00:33:44.002249 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.002136    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vlxfk_8389ea18-0b63-4ccf-b5ec-2e24b99b35fe/init-textfile/0.log"
Sep 10 00:33:44.196877 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.196845    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e6c34a8-c8da-4057-9e5d-c8877401f083/prometheus/0.log"
Sep 10 00:33:44.219753 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.219729    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e6c34a8-c8da-4057-9e5d-c8877401f083/config-reloader/0.log"
Sep 10 00:33:44.244415 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.244386    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e6c34a8-c8da-4057-9e5d-c8877401f083/thanos-sidecar/0.log"
Sep 10 00:33:44.274073 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.271303    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e6c34a8-c8da-4057-9e5d-c8877401f083/kube-rbac-proxy-web/0.log"
Sep 10 00:33:44.300202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.300170    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e6c34a8-c8da-4057-9e5d-c8877401f083/kube-rbac-proxy/0.log"
Sep 10 00:33:44.324486 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.324405    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e6c34a8-c8da-4057-9e5d-c8877401f083/kube-rbac-proxy-thanos/0.log"
Sep 10 00:33:44.347073 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.347024    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3e6c34a8-c8da-4057-9e5d-c8877401f083/init-config-reloader/0.log"
Sep 10 00:33:44.420230 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.420195    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-vs6h6_e6079083-3c7a-4020-ba22-599cf000e8d4/prometheus-operator-admission-webhook/0.log"
Sep 10 00:33:44.521831 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.521800    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74cd8cddb-tmzdc_e19cad74-ef52-400e-b684-b37fbbc770e0/thanos-query/0.log"
Sep 10 00:33:44.540929 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.540903    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74cd8cddb-tmzdc_e19cad74-ef52-400e-b684-b37fbbc770e0/kube-rbac-proxy-web/0.log"
Sep 10 00:33:44.579147 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.579071    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74cd8cddb-tmzdc_e19cad74-ef52-400e-b684-b37fbbc770e0/kube-rbac-proxy/0.log"
Sep 10 00:33:44.602324 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.602300    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74cd8cddb-tmzdc_e19cad74-ef52-400e-b684-b37fbbc770e0/prom-label-proxy/0.log"
Sep 10 00:33:44.625722 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.625692    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74cd8cddb-tmzdc_e19cad74-ef52-400e-b684-b37fbbc770e0/kube-rbac-proxy-rules/0.log"
Sep 10 00:33:44.651787 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:44.651755    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74cd8cddb-tmzdc_e19cad74-ef52-400e-b684-b37fbbc770e0/kube-rbac-proxy-metrics/0.log"
Sep 10 00:33:45.997226 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:45.997200    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-8jvvw_6543ed96-cfa4-4666-8547-ec6fd01de0c1/networking-console-plugin/0.log"
Sep 10 00:33:47.048612 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.048570    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"]
Sep 10 00:33:47.054655 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.054631    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.064237 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.064210    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"]
Sep 10 00:33:47.224189 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.224147    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztt6q\" (UniqueName: \"kubernetes.io/projected/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-kube-api-access-ztt6q\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.224394 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.224225    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-lib-modules\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.224394 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.224257    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-sys\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.224394 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.224309    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-proc\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.224589 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.224431    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-podres\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.325534 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.325455    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-podres\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.325534 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.325498    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztt6q\" (UniqueName: \"kubernetes.io/projected/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-kube-api-access-ztt6q\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.325534 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.325532    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-lib-modules\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.325815 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.325557    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-sys\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.325815 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.325596    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-proc\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.325815 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.325656    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-podres\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.325815 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.325717    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-proc\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.325815 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.325717    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-lib-modules\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.325815 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.325728    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-sys\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.337370 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.337338    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztt6q\" (UniqueName: \"kubernetes.io/projected/0417b711-bc20-4c47-b6d4-0b27c9fda2e1-kube-api-access-ztt6q\") pod \"perf-node-gather-daemonset-nb8l7\" (UID: \"0417b711-bc20-4c47-b6d4-0b27c9fda2e1\") " pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.368950 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.368915    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:47.665984 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:47.665913    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"]
Sep 10 00:33:48.395919 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:48.395875    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7" event={"ID":"0417b711-bc20-4c47-b6d4-0b27c9fda2e1","Type":"ContainerStarted","Data":"88779a4ff19ff747dff9d1eeaa1483ca5a661b63ab59f7b24db14bd066a1c561"}
Sep 10 00:33:48.396404 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:48.395946    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7" event={"ID":"0417b711-bc20-4c47-b6d4-0b27c9fda2e1","Type":"ContainerStarted","Data":"1873e7a21d6421630dfa350cc79b16376c5161b383b0bb4e7697768cd67934c0"}
Sep 10 00:33:48.396808 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:48.396767    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:48.412202 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:48.412146    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7" podStartSLOduration=1.4121253839999999 podStartE2EDuration="1.412125384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 00:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 00:33:48.409683473 +0000 UTC m=+2708.660819517" watchObservedRunningTime="2026-09-10 00:33:48.412125384 +0000 UTC m=+2708.663261428"
Sep 10 00:33:49.077786 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:49.077753    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zz486_a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2/dns/0.log"
Sep 10 00:33:49.095712 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:49.095678    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zz486_a8f6437f-cb3a-45e1-8dd8-6cb24482bcc2/kube-rbac-proxy/0.log"
Sep 10 00:33:49.161241 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:49.161209    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wj527_555a4739-f6bd-4762-9564-7593b7a4f04e/dns-node-resolver/0.log"
Sep 10 00:33:49.730050 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:49.730016    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l72vc_b62491be-c176-4206-9d7b-84caeae4c67d/node-ca/0.log"
Sep 10 00:33:51.157805 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:51.157779    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fbcb8d956-mkkn8_485f4554-94b3-4e40-ab8c-aed5f078bb14/kube-auth-proxy/0.log"
Sep 10 00:33:51.244212 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:51.244182    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-f5g58_a5f0c922-b35c-4fe8-8c67-2365c774e22f/payload-pre-processing/0.log"
Sep 10 00:33:52.168431 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:52.168387    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ctkq2_96cde171-d33b-4082-b727-bfd2d2574b7e/serve-healthcheck-canary/0.log"
Sep 10 00:33:52.761694 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:52.761660    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpflz_600c0165-b1e7-4a95-bbdb-7f3bcd7764bf/kube-rbac-proxy/0.log"
Sep 10 00:33:52.780262 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:52.780228    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpflz_600c0165-b1e7-4a95-bbdb-7f3bcd7764bf/exporter/0.log"
Sep 10 00:33:52.798851 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:52.798824    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpflz_600c0165-b1e7-4a95-bbdb-7f3bcd7764bf/extractor/0.log"
Sep 10 00:33:55.143591 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:55.143547    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f8584cb86-2sls6_3ee3f0da-1f3d-46b9-9194-e6424233972e/manager/0.log"
Sep 10 00:33:55.269659 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:55.269628    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-cz4kd_54466006-34a1-40ad-be74-b22d9ab53140/manager/0.log"
Sep 10 00:33:55.422257 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:55.422173    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2hczq/perf-node-gather-daemonset-nb8l7"
Sep 10 00:33:55.423239 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:55.423217    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cfd54796-hhlph_b105f9d5-9554-4055-9eb8-582aafbc0298/manager/0.log"
Sep 10 00:33:55.444753 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:55.444729    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cfd54796-hhlph_b105f9d5-9554-4055-9eb8-582aafbc0298/copy-manifests/0.log"
Sep 10 00:33:56.588124 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:56.588097    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5895cdb77f-tmnbn_a6fc01d1-f577-42bf-a1f4-decd0a3e2c6e/manager/0.log"
Sep 10 00:33:57.369277 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:57.369199    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fbcb8d956-mkkn8_485f4554-94b3-4e40-ab8c-aed5f078bb14/kube-auth-proxy/0.log"
Sep 10 00:33:57.433672 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:57.433637    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-f5g58_a5f0c922-b35c-4fe8-8c67-2365c774e22f/payload-pre-processing/0.log"
Sep 10 00:33:59.585957 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:59.585808    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fbcb8d956-mkkn8_485f4554-94b3-4e40-ab8c-aed5f078bb14/kube-auth-proxy/0.log"
Sep 10 00:33:59.657837 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:33:59.657802    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-f5g58_a5f0c922-b35c-4fe8-8c67-2365c774e22f/payload-pre-processing/0.log"
Sep 10 00:34:01.751876 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:01.751825    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fgzkl_1b6a789d-180e-4cbe-911d-820b513fb316/migrator/0.log"
Sep 10 00:34:01.769139 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:01.769116    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-fgzkl_1b6a789d-180e-4cbe-911d-820b513fb316/graceful-termination/0.log"
Sep 10 00:34:03.014210 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:03.014181    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2w8q5_0c59eba6-c8ab-4a3b-8c89-dad9d27a3922/kube-multus/0.log"
Sep 10 00:34:03.231340 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:03.231311    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dr7db_033a3818-5580-4cc4-9e17-a1d5fd0d6a71/kube-multus-additional-cni-plugins/0.log"
Sep 10 00:34:03.251073 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:03.251034    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dr7db_033a3818-5580-4cc4-9e17-a1d5fd0d6a71/egress-router-binary-copy/0.log"
Sep 10 00:34:03.269118 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:03.269049    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dr7db_033a3818-5580-4cc4-9e17-a1d5fd0d6a71/cni-plugins/0.log"
Sep 10 00:34:03.287021 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:03.287001    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dr7db_033a3818-5580-4cc4-9e17-a1d5fd0d6a71/bond-cni-plugin/0.log"
Sep 10 00:34:03.305285 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:03.305266    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dr7db_033a3818-5580-4cc4-9e17-a1d5fd0d6a71/routeoverride-cni/0.log"
Sep 10 00:34:03.323104 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:03.323084    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dr7db_033a3818-5580-4cc4-9e17-a1d5fd0d6a71/whereabouts-cni-bincopy/0.log"
Sep 10 00:34:03.340885 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:03.340861    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dr7db_033a3818-5580-4cc4-9e17-a1d5fd0d6a71/whereabouts-cni/0.log"
Sep 10 00:34:03.576514 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:03.576443    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-plj9c_21369195-b9f6-4653-ae29-d2a7a07eb1e2/network-metrics-daemon/0.log"
Sep 10 00:34:03.590795 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:03.590770    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-plj9c_21369195-b9f6-4653-ae29-d2a7a07eb1e2/kube-rbac-proxy/0.log"
Sep 10 00:34:04.401330 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:04.401303    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9kxr_f387b325-e109-4ea3-af39-59292992adb5/ovn-controller/0.log"
Sep 10 00:34:04.415598 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:04.415572    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9kxr_f387b325-e109-4ea3-af39-59292992adb5/ovn-acl-logging/0.log"
Sep 10 00:34:04.438138 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:04.438116    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9kxr_f387b325-e109-4ea3-af39-59292992adb5/kube-rbac-proxy-node/0.log"
Sep 10 00:34:04.460094 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:04.460075    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9kxr_f387b325-e109-4ea3-af39-59292992adb5/kube-rbac-proxy-ovn-metrics/0.log"
Sep 10 00:34:04.474403 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:04.474385    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9kxr_f387b325-e109-4ea3-af39-59292992adb5/northd/0.log"
Sep 10 00:34:04.501938 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:04.501917    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9kxr_f387b325-e109-4ea3-af39-59292992adb5/nbdb/0.log"
Sep 10 00:34:04.519993 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:04.519973    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9kxr_f387b325-e109-4ea3-af39-59292992adb5/sbdb/0.log"
Sep 10 00:34:04.583820 ip-10-0-139-211 kubenswrapper[2556]: I0910 00:34:04.583789    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k9kxr_f387b325-e109-4ea3-af39-59292992adb5/ovnkube-controller/0.log"